如何强制 dask worker 不将数据写入磁盘？

Question

我试图强制工人不要将数据写入磁盘，但可能我做错了什么。

我修改了文件~/.config/dask/distributed.yaml如下：

distributed:
  worker:
    # Fractions of worker memory at which we take action to avoid memory blowup
    # Set any of the lower three values to False to turn off the behavior entirely
    memory:
      target: 1.00  # target fraction to stay below
      spill: 1.00  # fraction at which we spill to disk
      pause: 1.00  # fraction at which we pause worker threads
      terminate: 1.00  # fraction at which we terminate the worker

但他们只是不断将任何临时结果写入磁盘。我也曾尝试使用虚构的内存限制设置 (100GB)，但它们仍然在任务结束时写入磁盘。我怎样才能强迫他们把所有的东西都留在记忆中？我错过了什么吗？

Answer 1

你正在做的应该没问题。我很惊讶它不适合你。

官方建议使用 false 值，如下所述：https://docs.dask.org/en/latest/setup/hpc.html#no-local-storage

distributed:
  worker:
    memory:
      target: false  # don't spill to disk
      spill: false  # don't spill to disk
      pause: 0.80  # pause execution at 80% memory use
      terminate: 0.95  # restart the worker at 95% use

如何强制 dask worker 不将数据写入磁盘？

How can I force a dask worker to not write data to disk?

dask

dask-distributed