具有 "spawn" 上下文的多处理无法访问 Linux 中的共享变量

Question

我必须在 Linux 中使用带有“spawn”上下文的 Process 方法。然后我写了一个示例代码如下：

from multiprocessing import Value
import multiprocessing

class Test(object):
    def __init__(self, m_val):
        print("step1")
        self.m_val = m_val
        print("step2")
        self.m_val_val = m_val.value
        self.prints()
    def prints(self):
        print("self.m_val_val:%d"%self.m_val_val)


def main(m_val):
    t = Test(m_val)

if __name__   == "__main__":
    N = 2
    procs = []
    v = Value("i",10)
    for i in range(0,N):
        proc_i = multiprocessing.get_context("spawn").Process(target=main,args=(v,))
        proc_i.daemon=True
        procs.append(proc_i)
    for i in range(0,N):
        procs[i].start()
    
    for i in range(0,N):
        procs[i].join()

当我在 Linux 中运行这段代码时，它会打印：

step1
step2
step1
step2

而在Windows中，打印内容将是：

step1
step2
self.m_val_val:10
step1
step2
self.m_val_val:10

此外，屏幕上也没有打印错误信息。那么，我该如何解决这个问题，即如何在 Linux?

中使用“spawn”上下文的同时在进程之间使用多处理值

Answer 1

问题是您在默认上下文中创建 Value，在 Unix 上是 fork。

您可以通过将默认启动上下文设置为 "spawn":

来解决此问题

multiprocessing.set_start_method("spawn")  # Add this
v = Value("i",10)

更好的是，在上下文中明确创建 Value：

# v = Value("i",10)                         # Change this
ctx = multiprocessing.get_context("spawn")  # to this
v = ctx.Value("i",10)                       #
for i in range(0,N):
    # proc_i = multiprocessing.get_context("spawn").Process(target=main,args=(v,))  # (Optional) Refactor this
    proc_i = ctx.Process(target=main,args=(v,))                                     # to this

参考

来自https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods：

spawn

Available on Unix and Windows. The default on Windows and macOS.

fork

Available on Unix only. The default on Unix.

Note that objects related to one context may not be compatible with processes for a different context. In particular, locks created using the fork context cannot be passed to processes started using the spawn or forkserver start methods.

在 macOS 上重现此问题

此问题可以在 Python < 3.8 的 macOS 上重现。

来自https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods：

Changed in version 3.8: On macOS, the spawn start method is now the default.

要在 Python 3.8 及更高版本的 macOS 上重现此问题：

multiprocessing.set_start_method("fork")  # Add this
v = Value("i",10)

错误消息和堆栈跟踪

OSError: [Errno 9] Bad file descriptor

Traceback (most recent call last):
  File "/path/to/python/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/path/to/python/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/path/to/file.py", line 18, in main
    t = Test(m_val)
  File "/path/to/file.py", line 10, in __init__
    self.m_val_val = m_val.value
  File "<string>", line 3, in getvalue
OSError: [Errno 9] Bad file descriptor

具有 "spawn" 上下文的多处理无法访问 Linux 中的共享变量

mutiprocessing with "spawn" context cannot access shared variables in Linux

python

python-multiprocessing

参考

在 macOS 上重现此问题

错误消息和堆栈跟踪