在 HIGHEST_PROTOCOL 实例化期间从 pickle 文件加载 class 实例

Question

目标

当我创建 ClassA 的实例时：
- 如果 pickle 文件存在，则从 pickle 文件加载实例
- 如果 pickle 文件不存在，则从头开始创建实例
使用 pickle HIGHEST_PROTOCOL.

失败

有人在评论中遇到同样的问题。问题也有描述：如果协议 > 1，则 pickle 在加载期间调用 __new__，这会产生无限递归。

# module_a.py
import os
import pickle
# import dill as pickle

save_path = r'C:\tests\pickle_tests\saved_instance_of_a.pkl'

def load(path):
    with open(path, 'rb') as f:
        return pickle.load(f)

def dump(x, path):
    with open(path, 'wb') as f:
        pickle.dump(
            x, f,
            protocol=pickle.HIGHEST_PROTOCOL)

class ClassA:
    def __new__(cls):
        print('__new__ called')

        if os.path.isfile(save_path):
            print('The saved pickle exists: loading from file.')
            instance = load(save_path)

        else:
            print('The saved pickle does not exist: creating.')
            instance = super(ClassA, cls).__new__(cls)

        return instance

    def __init__(self):
        print('__init__ called')
        if not os.path.isfile(save_path):
            self.my_dict = {'pi': 3.14}
            dump(self, save_path)

# myprogram.py
import os
import module_a

if __name__ =='__main__':
    instance_a = module_a.ClassA()
    print(instance_a.my_dict)

首先运行 OK（从头开始创建实例）：

$ python myprogram.py
__new__ called
The saved pickle does not exist: creating.
__init__ called
{'pi': 3.14}

第二次运行失败（从 pickle 加载实例）：

$ python myprogram.py
__new__ called
The saved pickle exists: loading from file.
__new__ called
The saved pickle exists: loading from file.
__new__ called
The saved pickle exists: loading from file.
__new__ called
The saved pickle exists: loading from file.
__new__ called
...
  File "C:\tests\pickle_tests\module_a.py", line 19, in __new__
    print('__new__ called')
  File "C:\anaconda\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
RecursionError: maximum recursion depth exceeded while calling a Python object

当前的解决方法

如果我替换上面的代码 protocol=pickle.HIGHEST_PROTOCOL 经过 protocol=0 但我不想使用协议 0（或 1）。我想使用 protocol=pickle.HIGHEST_PROTOCOL.

第一个运行好：

$ python myprogram.py
__new__ called
The saved pickle does not exist: creating.
__init__ called
{'pi': 3.14}

第二个运行好的：

$ python myprogram.py
__new__ called
The saved pickle exists: loading from file.
__init__ called
{'pi': 3.14}

Answer 1

由于 pickle 调用 __new__ 是导致问题的原因，允许使用 HIGHEST_PROTOCOL 的简单解决方法是让您不使用自己动手，在 __init__() 方法中完成所有操作。

这是一种方法：

myprogram.py:

import os
import module_a

if __name__ =='__main__':
    instance_a = module_a.ClassA()
    print(instance_a.my_dict)

module_a.py:

import os
import pickle

SAVE_PATH = r'C:\tests\pickle_tests\saved_instance_of_a.pkl'

def load(path):
    with open(path, 'rb') as f:
        return pickle.load(f)

def dump(x, path):
    with open(path, 'wb') as f:
        pickle.dump(x, f, protocol=pickle.HIGHEST_PROTOCOL)


class ClassA:
    def __init__(self):
        print('__init__ called')
        if os.path.isfile(SAVE_PATH):
            print('  loading from pickle file.')
            self.__dict__ = load(SAVE_PATH)
        else:
            print('  creating from scratch.')
            self.my_dict = {'pi': 3.14}
            dump(self.__dict__, SAVE_PATH)

在 HIGHEST_PROTOCOL 实例化期间从 pickle 文件加载 class 实例

Load class instance from pickle file during instantiation with HIGHEST_PROTOCOL

python

pickle

instantiation

目标

失败

当前的解决方法