Python 3.6+：嵌套的多处理管理器导致 FileNotFoundError

Question

所以我正在尝试在一个字典上使用多处理管理器，这是我最初的尝试：

from multiprocessing import Process, Manager

def task(stat):
    test['z'] += 1
    test['y']['Y0'] += 5


if __name__ == '__main__':
    test = Manager().dict({'x': {'X0': 10, 'X1': 20}, 'y': {'Y0': 0, 'Y1': 0}, 'z': 0})

    p = Process(target=task, args=(test,))
    p.start()
    p.join()

    print(test)

当然，当我运行这个时，输出不是我期望的，z 正确更新，而 y 没有变化！这是输出：

{'x': {'X0': 10, 'X1': 20}, 'y': {'Y0': 0, 'Y1': 0}, 'z': 1}

然后我用谷歌搜索，找到了一个解释 here，显然嵌套的字典也必须是 Manager().dict() 而不是正常的 python 字典（可能因为 Python 3.6).所以我做了以下事情：

from multiprocessing import Process, Manager

def task(stat):
    test['z'] += 1
    test['y']['Y0'] += 5


if __name__ == '__main__':
    test = Manager().dict({'x': Manager().dict({'X0': 10, 'X1': 20}), 'y': Manager().dict({'Y0': 0, 'Y1': 0}), 'z': 0})

    p = Process(target=task, args=(test,))
    p.start()
    p.join()

    print(test)
    print(test['y'])

但它没有正常工作，而是出现了这个无法解释的错误，为清楚起见，将其分为三个部分。第一部分对应于 test['y']['Y0'] += 5 而第二部分只是 print(test) 最后一部分是 print(test['y'])

的输出

Process Process-4:
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "shit.py", line 5, in task
    test['y']['Y0'] += 5
  File "<string>", line 2, in __getitem__
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 796, in _callmethod
    kind, result = conn.recv()
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 920, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 770, in __init__
    self._incref()
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 824, in _incref
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 492, in Client
    c = SocketClient(address)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 619, in SocketClient
    s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory




{'x': <DictProxy object, typeid 'dict' at 0x7f01de2c5860>, 'y': <DictProxy object, typeid 'dict' at 0x7f01de2c5898>, 'z': 1}




Traceback (most recent call last):
  File "test.py", line 16, in <module>
    print(test['y'])
  File "<string>", line 2, in __getitem__
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 796, in _callmethod
    kind, result = conn.recv()
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 920, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 770, in __init__
    self._incref()
  File "/usr/lib/python3.7/multiprocessing/managers.py", line 824, in _incref
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 492, in Client
    c = SocketClient(address)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 619, in SocketClient
    s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory

我不确定为什么会这样。内部字典显然已创建（如输出的第二部分所示）。但由于某种原因，它们根本无法读取或写入！。为什么会这样？

额外： 如果我通过 python 控制台（而不是脚本）运行相同的 python 代码，错误会从FileNotFoundError 到 ConnectionRefusedError。但具有相同的精确回溯！

Answer 1

在 Manager().dict() 中使用 Manager() 你每次都在开始一个新的 manager-process，所以你实际上是在嵌套管理器（如标题所说），这不是它应该的方式成为。相反，您需要做的是实例化 one 管理器，然后在该管理器实例上创建字典：

from multiprocessing import Process, Manager
from multiprocessing.managers import DictProxy


def task(test):  # use parameter `test`, else you rely on forking
    test['z'] += 1
    test['y']['Y0'] += 5


if __name__ == '__main__':

    with Manager() as m:

        test = m.dict({'x': m.dict({'X0': 10, 'X1': 20}),
                       'y': m.dict({'Y0': 0, 'Y1': 0}),
                       'z': 0})

        p = Process(target=task, args=(test,))
        p.start()
        p.join()

        print(test)
        print(test['y'])

        # convert to normal dict before closing manager for persistence
        # in parent or for printing dict behind proxies
        d = {k: dict(v) if isinstance(v, DictProxy) else v
             for k, v in test.items()}

    print(d) # Manager already closed here

示例输出：

{'x': <DictProxy object, typeid 'dict' at 0x7f98cdaaa588>, 'y': <DictProxy object, typeid 'dict' at 0x7f98cda99c50>, 'z': 1}
{'Y0': 5, 'Y1': 0}
{'x': {'X0': 10, 'X1': 20}, 'y': {'Y0': 5, 'Y1': 0}, 'z': 1}

Process finished with exit code 0

如果您计划从多个进程修改 manager-objects，您还需要使用 Manager.Lock。

Answer 2

当运行在 pycharm 中逐行执行 python 代码时，我遇到了这个问题。运行整个脚本解决了这个问题，旁边有绿色箭头：

if __name__ == '__main__':

出于某种原因 python 不喜欢运行逐行收集，但不确定为什么

Python 3.6+：嵌套的多处理管理器导致 FileNotFoundError

Python 3.6+: Nested multiprocessing managers cause FileNotFoundError

python

multiprocessing

python-3.x

python-multiprocessing

multiprocessing-manager