尝试通过 Python 中的多处理运行时出错 3

Question

下面的代码工作正常

  [process_data(item, data_frame_list[item]) for item in data_frame_list if data_frame_list[item].shape[0] > 5]

我正在尝试将此代码并行转换为运行

pool_obj = multiprocessing.Pool()
[pool_obj.map(process_data,item, data_frame_list[item]) for item in data_frame_list if data_frame_list[item].shape[0] > 5]

这会导致错误

Traceback (most recent call last):
      File "/home/pyuser/PycharmProjects/project_sample/testyard_2.py", line 425, in <module>
        [pool_obj.map(process_data,item, data_frame_list[item]) for item in data_frame_list if data_frame_list[item].shape[0] > 5]
      File "/home/pyuser/PycharmProjects/project_sample/testyard_2.py", line 425, in <listcomp>
        [pool_obj.map(process_data,item, data_frame_list[item]) for item in data_frame_list if data_frame_list[item].shape[0] > 5]
      File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
        return self._map_async(func, iterable, mapstar, chunksize).get()
      File "/usr/lib/python3.8/multiprocessing/pool.py", line 485, in _map_async
        result = MapResult(self, chunksize, len(iterable), callback,
      File "/usr/lib/python3.8/multiprocessing/pool.py", line 797, in __init__
        if chunksize <= 0:
      File "/home/pyuser/PycharmProjects/project_sample/venv/lib/python3.8/site-packages/pandas/core/ops/common.py", line 69, in new_method
        return method(self, other)
      File "/home/pyuser/PycharmProjects/project_sample/venv/lib/python3.8/site-packages/pandas/core/arraylike.py", line 44, in __le__
        return self._cmp_method(other, operator.le)
      File "/home/pyuser/PycharmProjects/project_sample/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 6849, in _cmp_method
        new_data = self._dispatch_frame_op(other, op, axis=axis)
      File "/home/pyuser/PycharmProjects/project_sample/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 6888, in _dispatch_frame_op
        bm = self._mgr.apply(array_op, right=right)
      File "/home/pyuser/PycharmProjects/project_sample/venv/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 325, in apply
        applied = b.apply(f, **kwargs)
      File "/home/pyuser/PycharmProjects/project_sample/venv/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 382, in apply
        result = func(self.values, **kwargs)
      File "/home/pyuser/PycharmProjects/project_sample/venv/lib/python3.8/site-packages/pandas/core/ops/array_ops.py", line 284, in comparison_op
        res_values = comp_method_OBJECT_ARRAY(op, lvalues, rvalues)
      File "/home/pyuser/PycharmProjects/project_sample/venv/lib/python3.8/site-packages/pandas/core/ops/array_ops.py", line 73, in comp_method_OBJECT_ARRAY
        result = libops.scalar_compare(x.ravel(), y, op)
      File "pandas/_libs/ops.pyx", line 107, in pandas._libs.ops.scalar_compare
    TypeError: '<=' not supported between instances of 'str' and 'int'

我无法找出我所做的不正确的地方。我可以请求一些指导吗？

Answer 1

使用了更容易使用的不同库。现在一切正常。

from joblib import Parallel, delayed
import multiprocessing

Parallel(n_jobs=multiprocessing.cpu_count())(delayed(process_data)(item, data_frame_list[item]) for item in data_frame_list if data_frame_list[item].shape[0] > 5)

尝试通过 Python 中的多处理运行时出错 3

Error when trying to run via multiprocessing in Python 3

python-3.x

python-multiprocessing

尝试通过 Python 中的多处理 运行 时出错 3

Error when trying to run via multiprocessing in Python 3

python-3.x

python-multiprocessing

尝试通过 Python 中的多处理运行时出错 3