如何在 python 中进行多线程上传？

Question

在python中我有一个需要上传的文件列表。

示例代码可能如下所示：

        for path, key in upload_list:
            upload_function.upload_file(path, key)

如何对此类任务进行多线程处理？

我已经遇到过多种解决方案，例如流程循环，请参阅：但对我来说，这些似乎有点矫枉过正，只是为了处理要上传的文件列表。有没有更聪明的方法？

Answer 1

ThreadPoolExecutor

没那么复杂

def upload_files(upload_list):
    def __upload(val):
        upload_function.upload_file(*val)

    with ThreadPoolExecutor() as executor:
        executor.map(__upload, upload_list)

upload_files(upload_list)

如果您可以修改 upload_function.upload_file 以接收 key 和 path 作为一个参数，则函数可以简化为

upload_list = [...]
with ThreadPoolExecutor() as executor:
    executor.map(upload_function.upload_file, upload_list)

executor.map(__upload, upload_list) 将运行 __upload 函数以 upload_list 中的一个子列表作为参数，并且它将运行所有这些（几乎）同时发生.

要控制线程数，您可以使用 max_workers 参数

with ThreadPoolExecutor(max_workers=len(upload_list)) as executor:

如何在 python 中进行多线程上传？

How to multithread uploads in python?

python

multithreading