如何使用 ThreadPoolExecutor 跟踪输入参数和结果

Question

假设我有以下代码：

    url_list = [

    "https://via.placeholder.com/480",
    "https://via.placeholder.com/490",
    "https://via.placeholder.com/500",
    "https://via.placeholder.com/510",
    "https://via.placeholder.com/520",
    "https://via.placeholder.com/530",
]


def download_file(url):
    html = requests.get(url, stream=True)
    return html.status_code

result = {} 
processes = []
with ThreadPoolExecutor(max_workers=10) as executor:
    for url in url_list:
        processes.append(executor.submit(download_file, url))

for task in as_completed(processes):
    result.update({url: task.result()})

我想存储在 url 的结果映射中：结果，类似于

https://via.placeholder.com/480 : 200
https://via.placeholder.com/490 : 200

但结果，我有以下

https://via.placeholder.com/530 : 200
https://via.placeholder.com/530 : 200
https://via.placeholder.com/530 : 200
https://via.placeholder.com/530 : 200
https://via.placeholder.com/530 : 200
https://via.placeholder.com/530 : 200
https://via.placeholder.com/530 : 200
https://via.placeholder.com/530 : 200

Answer 1

首先，如果您要访问同一个网站，使用 requests.Session 实例进行检索绝对值得。其次，您可以将 submit 返回的 Future 实例作为键存储在字典中，并将用作输入参数的 URL 作为值存储。我冒昧地重命名了一些变量，以便它们更接近于它们所持有的内容：

from concurrent.futures import ThreadPoolExecutor, as_completed
import requests

url_list = [

    "https://via.placeholder.com/480",
    "https://via.placeholder.com/490",
    "https://via.placeholder.com/500",
    "https://via.placeholder.com/510",
    "https://via.placeholder.com/520",
    "https://via.placeholder.com/530",
]

def download_file(session, url):
    html = session.get(url, stream=True)
    return html.status_code

result = {}
with requests.Session() as session:
    with ThreadPoolExecutor(max_workers=10) as executor:
        futures = {executor.submit(download_file, session, url): url for url in url_list}
        for future in as_completed(futures):
            # recover original URL:
            url = futures[future]
            result[url] = future.result()
print(result)

如何使用 ThreadPoolExecutor 跟踪输入参数和结果

How to track input parameter and result using ThreadPoolExecutor

python

threadpoolexecutor