执行 p.start() 后多处理抛出运行时错误
Multiprocessing throwing Runtime Error after executing p.start()
我的代码在 python 中执行 Multiprocessing 包的 p.start() 方法后出现运行时错误。
记录的错误如下：
Traceback (most recent call last): File
"/Users/anujpanchal/Documents/exportify-micro-booking/functionality/Notifications.py",
line 396, in send_notification
p.start() File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py",
line 121, in start
self._popen = self._Popen(self) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py",
line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj) File
"/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py",
line 284, in _Popen
return Popen(process_obj) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py",
line 32, in init
super().init(process_obj) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py",
line 19, in init
self._launch(process_obj) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py",
line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py",
line 154, in get_preparation_data
_check_not_importing_main() File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py",
line 134, in _check_not_importing_main
raise RuntimeError(''' RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "", line 1, in
File
"/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py",
line 116, in spawn_main
exitcode = _main(fd, parent_sentinel) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py",
line 125, in _main
prepare(preparation_data) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py",
line 236, in prepare
_fixup_main_from_path(data['init_main_from_path']) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py",
line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path, File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py",
line 268, in run_path
return _run_module_code(code, init_globals, run_name, File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py",
line 97, in _run_module_code
_run_code(code, mod_globals, init_globals, File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py",
line 87, in _run_code
exec(code, run_globals) File "/Users/anujpanchal/Documents/exportify-micro-booking/functionality/Notifications.py",
line 513, in
Notifications(checkpoint="discharged_on_transshipment").send_notification(booking_id="XPF100283",
tracking_obj=tracking_obj, scraper_update=False) File
"/Users/anujpanchal/Documents/exportify-micro-booking/functionality/Notifications.py",
line 470, in send_notification
return response.errorResponse("Some error occurred please try again!") File
"/Users/anujpanchal/Documents/exportify-micro-booking/functionality/response.py",
line 50, in errorResponse
return jsonify(response), status File "/Users/anujpanchal/Documents/exportify-micro-booking/venv/lib/python3.9/site-packages/flask/json/init.py",
line 339, in jsonify
if current_app.config['JSONIFY_PRETTYPRINT_REGULAR'] or current_app.debug: File
"/Users/anujpanchal/Documents/exportify-micro-booking/venv/lib/python3.9/site-packages/werkzeug/local.py",
line 348, in getattr
return getattr(self._get_current_object(), name) File "/Users/anujpanchal/Documents/exportify-micro-booking/venv/lib/python3.9/site-packages/werkzeug/local.py",
line 307, in _get_current_object
return self.__local() File "/Users/anujpanchal/Documents/exportify-micro-booking/venv/lib/python3.9/site-packages/flask/globals.py",
line 51, in _find_app
raise RuntimeError(_app_ctx_err_msg) RuntimeError: Working outside of application context.
This typically means that you attempted to use functionality that
needed to interface with the current application object in some way.
To solve this, set up an application context with app.app_context().
See the documentation for more information.
为什么会抛出这种类型的运行时错误，是因为我将 MacBook 更新到 Big Sur 11.3 还是因为其他原因？
我对 Mac 了解不多。一般来说，我相信大多数版本使用（或曾经使用）OS 调用 fork 来创建新进程。这意味着您不必将创建新进程的代码放在块中，例如 ...
if __name__ == '__main__':
... 使用 OS 的平台需要调用 spawn 来创建新进程，原因是当使用 spawn 时，新进程是通过初始化创建的一个新地址 space，启动一个新的 Python 解释器并重新 运行 从程序最顶部开始的代码，在全局范围内重新执行代码。这将导致在递归循环 ad infinitum 中重新执行创建新进程的代码，如果该代码没有被上述 if 语句有条件地执行的话。
在您对 Big Sur 11.3 的更新中，创建新进程的新默认方法似乎已从 fork 切换到 spawn -- 并且可能有充分的理由。所以我犹豫建议你先尝试打电话给...
multiprocessig.set_start_method('fork')
...恢复 fork 作为创建新进程的默认方法。
相反，我建议您将流程创建代码包含在上述 if 块中。很遗憾您决定不 post 任何代码，否则我可以向您展示具体的操作方法。
我在 运行 以下代码时遇到了同样的问题：
import multiprocessing as mp
import time
from multiprocessing.sharedctypes import Value
def worker(sema, pool, resource):
name = mp.current_process().name
print("{}: Making pool active".format(name))
with sema:
pool.make_active(name)
print("{}: Waiting for 1 second".format(name))
time.sleep(1)
print("{}: Modifying resource".format(name))
resource.value = resource.value + 1
print("{}: Making pool inactive".format(name))
pool.make_inactive(name)
class ActivePool(object):
def __init__(self):
super(ActivePool, self).__init__()
self.manager = mp.Manager()
self.active = self.manager.list()
self.lock = mp.Lock()
def make_active(self, name):
with self.lock:
self.active.append(name)
def make_inactive(self, name):
with self.lock:
self.active.remove(name)
if __name__ == "__main__":
mp.set_start_method('fork')
pool = ActivePool()
sema = mp.Semaphore(3)
resource = Value('d', 10.0)
procs = []
for i in range(20):
proc_name = "process_{}".format(i)
p = mp.Process(target=worker, args=(sema, pool, resource,), name=proc_name)
procs.append(p)
for p in procs:
p.start()
for p in procs:
p.join()
print("Value of resource after modification: {}".format(resource.value))
以上代码在 if 循环中添加以下内容后有效：
multiprocessig.set_start_method('fork')
我的代码在 python 中执行 Multiprocessing 包的 p.start() 方法后出现运行时错误。
记录的错误如下：
Traceback (most recent call last): File "/Users/anujpanchal/Documents/exportify-micro-booking/functionality/Notifications.py", line 396, in send_notification p.start() File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen return Popen(process_obj) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in init super().init(process_obj) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in init self._launch(process_obj) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch prep_data = spawn.get_preparation_data(process_obj._name) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 154, in get_preparation_data _check_not_importing_main() File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 134, in _check_not_importing_main raise RuntimeError(''' RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable.During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "", line 1, in File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 125, in _main prepare(preparation_data) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 268, in run_path return _run_module_code(code, init_globals, run_name, File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/anujpanchal/Documents/exportify-micro-booking/functionality/Notifications.py", line 513, in Notifications(checkpoint="discharged_on_transshipment").send_notification(booking_id="XPF100283", tracking_obj=tracking_obj, scraper_update=False) File "/Users/anujpanchal/Documents/exportify-micro-booking/functionality/Notifications.py", line 470, in send_notification return response.errorResponse("Some error occurred please try again!") File "/Users/anujpanchal/Documents/exportify-micro-booking/functionality/response.py", line 50, in errorResponse return jsonify(response), status File "/Users/anujpanchal/Documents/exportify-micro-booking/venv/lib/python3.9/site-packages/flask/json/init.py", line 339, in jsonify if current_app.config['JSONIFY_PRETTYPRINT_REGULAR'] or current_app.debug: File "/Users/anujpanchal/Documents/exportify-micro-booking/venv/lib/python3.9/site-packages/werkzeug/local.py", line 348, in getattr return getattr(self._get_current_object(), name) File "/Users/anujpanchal/Documents/exportify-micro-booking/venv/lib/python3.9/site-packages/werkzeug/local.py", line 307, in _get_current_object return self.__local() File "/Users/anujpanchal/Documents/exportify-micro-booking/venv/lib/python3.9/site-packages/flask/globals.py", line 51, in _find_app raise RuntimeError(_app_ctx_err_msg) RuntimeError: Working outside of application context.
This typically means that you attempted to use functionality that needed to interface with the current application object in some way. To solve this, set up an application context with app.app_context(). See the documentation for more information.
为什么会抛出这种类型的运行时错误，是因为我将 MacBook 更新到 Big Sur 11.3 还是因为其他原因？
我对 Mac 了解不多。一般来说，我相信大多数版本使用（或曾经使用）OS 调用 fork 来创建新进程。这意味着您不必将创建新进程的代码放在块中，例如 ...
if __name__ == '__main__':
... 使用 OS 的平台需要调用 spawn 来创建新进程，原因是当使用 spawn 时，新进程是通过初始化创建的一个新地址 space，启动一个新的 Python 解释器并重新 运行 从程序最顶部开始的代码，在全局范围内重新执行代码。这将导致在递归循环 ad infinitum 中重新执行创建新进程的代码，如果该代码没有被上述 if 语句有条件地执行的话。
在您对 Big Sur 11.3 的更新中，创建新进程的新默认方法似乎已从 fork 切换到 spawn -- 并且可能有充分的理由。所以我犹豫建议你先尝试打电话给...
multiprocessig.set_start_method('fork')
...恢复 fork 作为创建新进程的默认方法。
相反，我建议您将流程创建代码包含在上述 if 块中。很遗憾您决定不 post 任何代码，否则我可以向您展示具体的操作方法。
我在 运行 以下代码时遇到了同样的问题：
import multiprocessing as mp
import time
from multiprocessing.sharedctypes import Value
def worker(sema, pool, resource):
name = mp.current_process().name
print("{}: Making pool active".format(name))
with sema:
pool.make_active(name)
print("{}: Waiting for 1 second".format(name))
time.sleep(1)
print("{}: Modifying resource".format(name))
resource.value = resource.value + 1
print("{}: Making pool inactive".format(name))
pool.make_inactive(name)
class ActivePool(object):
def __init__(self):
super(ActivePool, self).__init__()
self.manager = mp.Manager()
self.active = self.manager.list()
self.lock = mp.Lock()
def make_active(self, name):
with self.lock:
self.active.append(name)
def make_inactive(self, name):
with self.lock:
self.active.remove(name)
if __name__ == "__main__":
mp.set_start_method('fork')
pool = ActivePool()
sema = mp.Semaphore(3)
resource = Value('d', 10.0)
procs = []
for i in range(20):
proc_name = "process_{}".format(i)
p = mp.Process(target=worker, args=(sema, pool, resource,), name=proc_name)
procs.append(p)
for p in procs:
p.start()
for p in procs:
p.join()
print("Value of resource after modification: {}".format(resource.value))
以上代码在 if 循环中添加以下内容后有效：
multiprocessig.set_start_method('fork')