从 pickle 文件加载 multiprocessing.manager.dictionary 时出错

Question

从 pickle 加载 multiprocessing.manager.dictionary 时出现错误。我确定 pickle 文件存在于我运行 python.

的目录中

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> import multiprocessing
>>> a = open("test.pkl", "rb")
>>> pickle.load(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 879, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 733, in __init__
    self._incref()
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 783, in _incref
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client
    c = SocketClient(address)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 304, in SocketClient
    s.connect(address)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 2] No such file or directory
>>> pickle.load(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1165, in load_put
    self.memo[self.readline()[:-1]] = self.stack[-1]
IndexError: list index out of range

这是将其保存到泡菜中的部分代码。

from multiprocessing import Manager, Pool
import pickle
from functools import partial

def output(dic, s):
    a = open(s + ".pkl", "wb")
    pickle.dump(dic, a)
    a.close()

data_list = [1,2,3,4]
pool = Pool(processes = 4)
m = Manager()
lock = m.Lock()
dic1 = m.dict()
func = partial(f, dic1) # f is a function that takes 3 arguments with the returned result stored in dic1
pool.map(func, data_list)
output(dic1, "test")
pool.close()
pool.join()

如果我在输出之前打印字典，终端上的结果看起来不错。

Answer 1

Manager.dict() 实际上 return 不是字典，而是管理器在不同进程中管理的真实字典的代理对象。当您对该对象调用方法时，该调用将转发到实际字典所在的管理器。

由于管理器和客户端进程之间的通信是使用 pickle 协议进行的，因此这些对象是可腌制的，并且当取消腌制时会再次生成指向管理器进程内部实际对象的代理，这就是为什么您会看到尝试创建一个当您尝试解开 dict 时的连接。

因此，如果您想 pickle 经理口述的内容，请先将其转换为实际口述：

...
pickle.dump(dict(dic), a)
...

从 pickle 文件加载 multiprocessing.manager.dictionary 时出错

Error while loading multiprocessing.manager.dictionary from pickle file

python

python-multiprocessing