Python memory_profiler: @profile 不支持多线程

Question

我在 example 文件夹中有以下代码，但我添加了 @profile。我只是想做这个例子运行因为在我更复杂的代码中我有同样的错误，我想知道每一行使用了多少内存。

系统：

Python: 3.9

内存分析器：0.58

OS：曼扎罗

代码：

import time
import multiprocessing as mp
from memory_profiler import profile
# Big numbers
X6 = 10 ** 6
X7 = 10 ** 7


def worker(num, wait, amt=X6):
    """
    A function that allocates memory over time.
    """
    frame = []

    for idx in range(num):
        frame.extend([1] * amt)
        time.sleep(wait)

    del frame


def main_sequential():
    """
    A sequential version of the work, where one worker is called at a time.
    """
    worker(5, 5, X6)
    worker(5, 2, X7)
    worker(5, 5, X6)
    worker(5, 2, X7)

@profile
def main_multiproc():
    """
    A multiprocessing version of the work, where workers work in their own
    child processes and are collected by the master process.
    """
    pool = mp.Pool(processes=4)
    tasks = [
        pool.apply_async(worker, args) for args in
        [(5, 5, X6), (5, 2, X7), (5, 5, X6), (5, 2, X7)]
    ]

    results = [p.get() for p in tasks]


if __name__ == '__main__':
    main_multiproc()

运行命令：

python -m memory_profiler MyScript.py

错误：


Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/vlad/Documents/TradingBotRL/venv/lib/python3.9/site-packages/memory_profiler.py", line 1303, in <module>
    exec_with_profiler(script_filename, prof, args.backend, script_args)
  File "/home/vlad/Documents/TradingBotRL/venv/lib/python3.9/site-packages/memory_profiler.py", line 1204, in exec_with_profiler
    exec(compile(f.read(), filename, 'exec'), ns, ns)
  File "MyScript.py", line 47, in <module>
    main_multiproc()
  File "/home/vlad/Documents/TradingBotRL/venv/lib/python3.9/site-packages/memory_profiler.py", line 1142, in wrapper
    val = prof(func)(*args, **kwargs)
  File "/home/vlad/Documents/TradingBotRL/venv/lib/python3.9/site-packages/memory_profiler.py", line 717, in f
    return func(*args, **kwds)
  File "MyScript.py", line 43, in main_multiproc
    results = [p.get() for p in tasks]
  File "MyScript.py", line 43, in <listcomp>
    results = [p.get() for p in tasks]
  File "/usr/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/lib/python3.9/multiprocessing/pool.py", line 537, in _handle_tasks
    put(task)
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 211, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.9/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function worker at 0x5556de78daf0>: attribute lookup worker on __main__ failed

我是不是漏掉了什么？

Answer 1

memory_profiler 的文档：https://pypi.org/project/memory-profiler/ 如果您使用装饰器 (@profile)，请说以下内容：

In this case the script can be run without specifying -m memory_profiler in the command line.

所以我认为你只需要运行 python MyScript.py

Python memory_profiler: @profile 不支持多线程

Python memory_profiler: @profile not working on multithreading

python

memory

memory-leaks

memory-profiling

python-3.x