Tqdm 进度条仅在 ProcessPoolExecutor 进程结束后显示

Question

我的 TQDM 进度条在我的多线程进程中不显示，我只有在进程完成后才能看到它

这是重现问题的方法

我编码了这两个方法

from concurrent.futures import ProcessPoolExecutor
import sys
from colorama import Fore

def parallelize(desc, func, array, max_workers):
    with ProcessPoolExecutor(max_workers=max_workers) as executor:
        output_data = list(progress_bar(desc, list(executor.map(func,array))))
    return output_data

def progress_bar(desc, array):
    return tqdm(array,
            total=len(array),
            file=sys.stdout,
            ascii=' >',
            desc=desc,
            bar_format="%s{l_bar}%s{bar:30}%s{r_bar}" % (Fore.RESET, Fore.BLUE, Fore.RESET))

你可以这样测试

from tqdm import tqdm
  
test = range(int(1e4))
   
def identity(x):
    return x

parallelize("", identity, test, 2)

它应该打印这个 (00:00) 但这个过程大约需要 3sc

100%|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>| 10000/10000 [00:00<00:00, 3954279.25it/s]

感谢您的帮助

Answer 1

我认为这是你调用进度条的原因

output_data = list(progress_bar(desc, list(executor.map(func,array))))

python 先 executor.map(func, array) 然后才将结果传递给 progress_bar。它不会相同，但我可以与您分享如何并行化 python 函数的样板。

from joblib import Parallel, delayed

def func(a):
  # Do something

# Parallelize the call
Parallel(n_jobs=-1)(delayed(func)(a) for a in tqdm(array, total=len(array))

Answer 2

替换了这个方法并且有效

def parallelize(desc, func, array, max_workers):
    return Parallel(n_jobs=max_workers)(delayed(func)(a) for a in progress_bar(desc, array))

Tqdm 进度条仅在 ProcessPoolExecutor 进程结束后显示

Tqdm progress bar only shows after process end with ProcessPoolExecutor

python

multiprocess

tqdm