使用 lambda 序列化 cython class 时出现 pickle 错误

Question

我使用 pickle 和 dill 来执行 lambda 函数并且工作正常:

import dill
import pickle

f = lambda x,y: x+y
s = pickle.dumps(f)

甚至在 class 中使用时，例如：

文件

foo.py

class Foo(object):
    def __init__(self):
        self.f = lambda x, y: x+y

文件

test.py

import dill
import pickle
from foo import Foo

f = Foo()
s = pickle.dumps(f)  # or  s = dill.dumps(f)

但是当使用 cython 构建格式为 .pyx (foo.pyx) 的相同文件时，无法使用 dill、pickle 或 cpickle 进行序列化，出现此错误:

Traceback (most recent call last): File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_cod exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in a = pickle.dumps(c) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 1380, in dumps Pickler(file, protocol).dump(obj) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 224, in dump self.save(obj) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 425, in save_reduce save(state) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/site-packages/dill/_dill.py", line 912, in save_module_dict StockPickler.save_dict(pickler, obj) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 655, in save_dict self._batch_setitems(obj.iteritems()) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 669, in _batch_setitems save(v) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 317, in save self.save_global(obj, rv) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 754, in save_global (obj, module, name)) PicklingError: Can't pickle . at 0x7f9ab1ff07d0>: it's not found as foo.lambda

setup.py 用于构建 cython 的文件

setup.py

from distutils.core import setup
from Cython.Build import cythonize

setup(ext_modules=cythonize("foo.pyx"))

然后在终端中运行：

python setup.py build_ext --inplace

有办法吗？

Answer 1

在此代码中

import dill
import pickle

f = lambda x,y: x+y
s = pickle.dumps(f)

f 是一个函数，但是在另一个代码中

import dill
import pickle
from foo import Foo

f = Foo()
s = pickle.dumps(f)  
# or  
s = dill.dumps(f)

f 是一个 class

Answer 2

我是 dill 作者。扩展@DavidW 在评论中所说的内容——我相信（目前）没有已知的序列化程序可以 pickle cython lambdas 或绝大多数 cython 代码。事实上，python 序列化程序要能够 pickle 具有 C 扩展的对象要困难得多，除非 C 扩展代码的作者专门构建序列化指令（如 numpy 和 pandas).在这种情况下......而不是lambda，你可以用__call__方法构建一个class，所以它就像一个函数......然后添加一个或多个泡菜方法（__reduce__、__getstate__、__setstate__ 或类似的东西）... 然后您应该能够 pickle class 的实例。这有点工作，但由于这条路径已被用来腌制用 C++ 编写的 classes——我相信你应该能够让它为 cython 构建的 classes 工作。

使用 lambda 序列化 cython class 时出现 pickle 错误

pickle error when serialize cython class with lambda

python

lambda

cython

pickle

dill