PyCharm 中的多处理导致 _pickle.PicklingError

Question

如何在 PyCharm Python 控制台（科学/交互模式）中使用 multiprocessing.Pool 而不会导致 _pickle.PicklingError ？

IDE

PyCharm 2021.3.2 (Professional Edition)
Build #PY-213.6777.50, built on January 27, 2022
Licensed to Trenton J. McKinney / Trenton McKinney
Subscription is active until February 27, 2022.
Runtime version: 11.0.13+7-b1751.25 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
Linux 3.10.0-1160.49.1.el7.x86_64
GC: G1 Young Generation, G1 Old Generation
Memory: 2048M
Cores: 32
Current Desktop: XFCE

代码

以下代码适用于终端和 Jupyter Lab，但不适用于 PyCharm 2021.3.2
这是一个简单的可重现示例

from multiprocessing import Pool


def f(x):
    return x*x


if __name__ == '__main__':
    with Pool(32) as p:
        print(p.map(f, range(32)))

错误

/home/tmckinney/anaconda3/bin/python3.9 /home/tmckinney/opt/pycharm-2021.3.1/plugins/python/helpers/pydev/pydevconsole.py --mode=client --port=5829
import sys; print('Python %s on %s' % (sys.version, sys.platform))
sys.path.extend(['/home/tmckinney/Documents/GitHub/hap_ingest'])
Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.29.0 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.29.0
Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
[GCC 7.5.0] on linux
runfile('/home/tmckinney/Documents/GitHub/hap_ingest/py_files/multiprocessingTest.py', wdir='/home/tmckinney/Documents/GitHub/hap_ingest/py_files')
Traceback (most recent call last):
  File "/home/tmckinney/anaconda3/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-8b3da05130fe>", line 1, in <module>
    runfile('/home/tmckinney/Documents/GitHub/hap_ingest/py_files/multiprocessingTest.py', wdir='/home/tmckinney/Documents/GitHub/hap_ingest/py_files')
  File "/home/tmckinney/opt/pycharm-2021.3.1/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/tmckinney/opt/pycharm-2021.3.1/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/tmckinney/Documents/GitHub/hap_ingest/py_files/multiprocessingTest.py", line 10, in <module>
    print(p.map(f, range(32)))
  File "/home/tmckinney/anaconda3/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/tmckinney/anaconda3/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/tmckinney/anaconda3/lib/python3.9/multiprocessing/pool.py", line 537, in _handle_tasks
    put(task)
  File "/home/tmckinney/anaconda3/lib/python3.9/multiprocessing/connection.py", line 211, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/tmckinney/anaconda3/lib/python3.9/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function f at 0x7f9ce1befa60>: attribute lookup f on __main__ failed

预期输出

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961]

研究

如其他文档所述，Windows 中 multiprocessing 模块的交互使用存在一些问题。这不是这个问题要解决的问题。

Answer 1

这个 bug 似乎有两个解决方法，已在 Windows 和 Linux 上测试：

这似乎是 PyCharm > 2021.1.3 的问题
- 仅在专业版中测试
- 未针对低于 2021.1.3 的版本进行测试
PY-50116 Multiprocessing package functions won't run in the interactive Python Console after upgrade to PyCharm 2021.2
PY-20885 Execution of multiprocessing.Pool gives an exception in Python Console
PY-52857 multiprocessing in PyCharm results in _pickle.PicklingError

如果需要使用 Python 控制台

使用科学模式进行探索性分析可能就是这种情况
解决方法是使用PyCharm 2021.1.3
- 当 Python 控制台
- - 看到 multiprocessing 在 PyCharm 2021.1.3 中可以在交互/科学模式下正常工作。

如果不需要使用 Python 控制台

取消勾选Run with Python Console

进行 Run/Debug 配置

可以通过以下方式之一访问编辑配置

Answer 2

在 Windows 中不可能。您正在尝试运行无法在交互模式下工作的代码。但结果 PyCharm 运行将代码作为脚本，这是一个错误。

来自 Python 文档：

此包中的功能要求 __main__ 模块可由 children 导入。这在编程指南中有所介绍，但值得在此指出。这意味着某些示例，例如 multiprocessing.pool.Pool 示例将无法在交互式解释器中运行。

https://docs.python.org/3/library/multiprocessing.html#multiprocessing-programming

PyCharm 中的多处理导致 _pickle.PicklingError

multiprocessing in PyCharm results in _pickle.PicklingError

multiprocessing

pycharm

python-3.x

IDE

代码

错误

预期输出

研究

如果需要使用 Python 控制台

如果不需要使用 Python 控制台

进行 Run/Debug 配置