在 dask 中为 LocalCluster 设置内存参数的正确方法

Question

我尝试了下面的代码，

from dask.distributed import Client, LocalCluster
worker_kwargs = {
    'memory_limit': '2G',
    'memory_target_fraction': 0.6,
    'memory_spill_fraction': 0.7,
    'memory_pause_fraction': 0.8,
    'memory_terminate_fraction': 0.95,
}
cluster = LocalCluster(ip='0.0.0.0', n_workers=8, **worker_kwargs)

但是没用

...
TypeError: __init__() got an unexpected keyword argument 'memory_target_fraction'

正确的打电话方式是什么？ LocalCluster 似乎也不关心 ~/.config/dask/distributed.yaml。

我读过这些帖子：

1. https://github.com/dask/distributed/issues/2456
2. https://github.com/dask/distributed/issues/1521

Answer 1

看来是版本问题。更新dask到1.2.0解决问题

from dask.distributed import Client, LocalCluster
worker_kwargs = {
    'memory_limit': '2G',
    'memory_target_fraction': 0.6,
    'memory_spill_fraction': 0.7,
    'memory_pause_fraction': 0.8,
    # 'memory_terminate_fraction': 0.95,
}
cluster = LocalCluster(ip='0.0.0.0', n_workers=8, **worker_kwargs)

有效