连接到 dask.distributed 集群时出现 Pickle 错误
Pickle error when connecting to dask.distributed cluster
这是我简单的 code.Trying 到 运行 我的第一个程序。
from dask.distributed import Client
client = Client('192.168.1.102:8786')
def inc(x):
return x + 1
x = client.submit(inc, 10)
print(x.result())
当尝试使用此命令 运行 此代码时:
$python3 filename.py
我收到这个错误:
/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py
- INFO - Failed to serialize <function inc at 0x7f678ad05840> Traceback (most recent call last): File
"/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py",
line 33, in dumps
return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL) AttributeError: 'module' object has no attribute 'dumps'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py",
line 43, in dumps
return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL) AttributeError: 'module' object has no attribute 'dumps' Traceback
(most recent call last): File
"/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py",
line 33, in dumps
return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL) AttributeError: 'module' object has no attribute 'dumps'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "dis.py", line 8, in
<module>
x = client.submit(inc, 10) File "/usr/local/lib/python3.4/dist-packages/distributed/client.py", line
643, in submit
loose_restrictions, priority={skey: 0}) File "/usr/local/lib/python3.4/dist-packages/distributed/client.py", line
1235, in _graph_to_futures
'tasks': valmap(dumps_task, dsk3), File "/usr/local/lib/python3.4/dist-packages/toolz/dicttoolz.py", line 84,
in valmap
rv.update(zip(iterkeys(d), map(func, itervalues(d)))) File "/usr/local/lib/python3.4/dist-packages/distributed/worker.py", line
812, in dumps_task
return {'function': dumps_function(task[0]), File "/usr/local/lib/python3.4/dist-packages/distributed/worker.py", line
779, in dumps_function
b = dumps(func) File "/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py",
line 43, in dumps
return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL) AttributeError: 'module' object has no attribute 'dumps'
通常 pickling 错误表明您的 Dask.distributed 网络的某些元素(工作者、调度程序或客户端)存在 Python 版本不匹配。也许您的工人 运行 低于 Python 2 而您却没有意识到?
那个错误特别指出 cloudpickle
库没有 dumps
方法,这很奇怪。据我所知,cloudpickle 一直有一个 dumps
函数。您的环境中是否有一个奇怪的 cloudpickle 库或一个非常旧的版本?
如果您只是尝试一下,您还可以通过省略调度程序的地址在同一进程中启动本地集群
from dask.distributed import Client
# client = Client('scheduler-address:8786')
client = Client() # create local "cluster"
这是我简单的 code.Trying 到 运行 我的第一个程序。
from dask.distributed import Client
client = Client('192.168.1.102:8786')
def inc(x):
return x + 1
x = client.submit(inc, 10)
print(x.result())
当尝试使用此命令 运行 此代码时:
$python3 filename.py
我收到这个错误:
/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py
- INFO - Failed to serialize <function inc at 0x7f678ad05840> Traceback (most recent call last): File
"/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py",
line 33, in dumps
return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL) AttributeError: 'module' object has no attribute 'dumps'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py",
line 43, in dumps
return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL) AttributeError: 'module' object has no attribute 'dumps' Traceback
(most recent call last): File
"/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py",
line 33, in dumps
return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL) AttributeError: 'module' object has no attribute 'dumps'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "dis.py", line 8, in
<module>
x = client.submit(inc, 10) File "/usr/local/lib/python3.4/dist-packages/distributed/client.py", line
643, in submit
loose_restrictions, priority={skey: 0}) File "/usr/local/lib/python3.4/dist-packages/distributed/client.py", line
1235, in _graph_to_futures
'tasks': valmap(dumps_task, dsk3), File "/usr/local/lib/python3.4/dist-packages/toolz/dicttoolz.py", line 84,
in valmap
rv.update(zip(iterkeys(d), map(func, itervalues(d)))) File "/usr/local/lib/python3.4/dist-packages/distributed/worker.py", line
812, in dumps_task
return {'function': dumps_function(task[0]), File "/usr/local/lib/python3.4/dist-packages/distributed/worker.py", line
779, in dumps_function
b = dumps(func) File "/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py",
line 43, in dumps
return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL) AttributeError: 'module' object has no attribute 'dumps'
通常 pickling 错误表明您的 Dask.distributed 网络的某些元素(工作者、调度程序或客户端)存在 Python 版本不匹配。也许您的工人 运行 低于 Python 2 而您却没有意识到?
那个错误特别指出 cloudpickle
库没有 dumps
方法,这很奇怪。据我所知,cloudpickle 一直有一个 dumps
函数。您的环境中是否有一个奇怪的 cloudpickle 库或一个非常旧的版本?
如果您只是尝试一下,您还可以通过省略调度程序的地址在同一进程中启动本地集群
from dask.distributed import Client
# client = Client('scheduler-address:8786')
client = Client() # create local "cluster"