如何从 dask 包中获取未来的对象？
How to get future object from dask bag?
我需要从未来调用回调函数来对任务执行重试逻辑。但是我不知道如何从 dask bag 中获取 future 对象。
例如dask bag是这样使用的
b = db.from_sequence(['1.dat', '2.dat', ...]).map(process_filename)
b.compute()
# I need a callback function when a task is finish with a single file to have complex retry logic
# I can get a future object from client.submit(my_args).add_done_callback(my_call_back) but I don't want to use client.submit because of large data set
您可以通过以下方式从袋子分区中获取期货：
from dask.distributed import futures_of
b = b.persist()
futures = futures_of(b)
我需要从未来调用回调函数来对任务执行重试逻辑。但是我不知道如何从 dask bag 中获取 future 对象。
例如dask bag是这样使用的
b = db.from_sequence(['1.dat', '2.dat', ...]).map(process_filename)
b.compute()
# I need a callback function when a task is finish with a single file to have complex retry logic
# I can get a future object from client.submit(my_args).add_done_callback(my_call_back) but I don't want to use client.submit because of large data set
您可以通过以下方式从袋子分区中获取期货：
from dask.distributed import futures_of
b = b.persist()
futures = futures_of(b)