使用Python asyncio corurrent send network request时，如何让coroutine优先继续处理response，而不是发送新的请求？

Question

如果我要请求API1000次（发送网络请求和处理响应），它会先发送所有1000次请求后开始处理响应，然后处理响应。

如果完成，我可以告诉 asyncio 更喜欢 return 返回等待位置代码吗？

import asyncio

import httpx

# request_time = 10
request_time = 1000 # large enough to ensure previous responses return

limits = httpx.Limits(max_connections=5)
client = httpx.AsyncClient(limits=limits)


async def request_baidu(i):
    # async with httpx.AsyncClient(limits=limits) as client:
        print(f"===> %d" % i)
        r = await client.get("http://www.baidu.com")
        # print(r.status_code)
        print(f"<=== %d" % i) # How to ensure return to run this code, not make a new request (run a new task `request_baidu` here)


async def main():
    request_list = [asyncio.create_task(request_baidu(i)) for i in range(request_time)]
    await asyncio.gather(*request_list)

if __name__ == '__main__':
    asyncio.run(main())

结果

# request_time = 10

===> 0
===> 1
===> 2
===> 3
===> 4
===> 5
===> 6
===> 7
===> 8
===> 9
<=== 3 # (we can see it continue to handle response after sending all request)
<=== 4
<=== 0 
<=== 1
<=== 2
<=== 5
<=== 6
<=== 7
<=== 8
<=== 9

预期结果：

===> 0
===> 1
<=== 0 #(continue handle response when there is some response)
===> 2
===> 3
<=== 1
<=== 2
===> 4
# ...

Answer 1

您可以使用asyncio.as_completed获得最早的下一个结果：

test.py:

import asyncio
import time

from httpx import AsyncClient, Limits

REQUESTS = 10
URL = "http://www.baidu.com"
TIMEOUT = 5


async def request_baidu(client, i):
    t1 = time.process_time()
    print(f"===> {i}")
    r = await client.get(f"{URL}", timeout=TIMEOUT)

    return f"<=== {i} ({time.process_time() - t1:.3f}s)"


async def main():
    async with AsyncClient() as client:
        aws = [asyncio.create_task(request_baidu(client, i)) for i in range(REQUESTS)]

        for coro in asyncio.as_completed(aws):
            earliest_result = await coro
            print(earliest_result)


if __name__ == "__main__":
    asyncio.run(main())

测试：

$ python test.py
===> 0
===> 1
===> 2
===> 3
===> 4
===> 5
===> 6
===> 7
===> 8
===> 9
<=== 9 (0.073s)
<=== 2 (0.081s)
<=== 1 (0.082s)
<=== 0 (0.086s)
<=== 8 (0.086s)
<=== 4 (0.088s)
<=== 6 (0.090s)
<=== 3 (0.092s)
<=== 7 (0.092s)
<=== 5 (0.093s)

使用Python asyncio corurrent send network request时，如何让coroutine优先继续处理response，而不是发送新的请求？

When using Python asyncio corurrent send network request, how to make coroutine prefer continue to handle response first, not send a new request?

python

web-crawler

python-asyncio

coroutine

httpx

结果

预期结果：