asyncio.gather() 在具有协程字段的字典列表上？

Question

我有以下两个异步函数

from tornado.httpclient import AsyncHTTPClient

async def get_categories(): # return a list of str
    # ....
    http = AsyncHTTPClient()
    resp = await http.fetch(....)
    return [....]

async def get_details(category): # return a list of dict
    # ....
    http = AsyncHTTPClient()
    resp = await http.fetch(....)
    return [....]

现在我需要创建一个函数来获取所有类别的详细信息（运行同时通过 http 获取）并将它们组合在一起。

async def get_all_details():
    categories = await get_categories()
    tasks = list(map(lambda x: {'category': x, 'task':get_details(x)}, categories))
    r = await asyncio.gather(*tasks) # error

# need to return [
#   {'category':'aaa', 'detail':'aaa detail 1'}, 
#   {'category':'aaa', 'detail':'aaa detail 2'}, 
#   {'category':'bbb', 'detail':'bbb detail 1'}, 
#   {'category':'bbb', 'detail':'bbb detail 2'}, 
#   {'category':'bbb', 'detail':'bbb detail 3'}, 
#   {'category':'ccc', 'detail':'ccc detail 1'}, 
#   {'category':'ccc', 'detail':'aaa detail 2'}, 
# ]

然而，列表行return错误：

TypeError: unhashable type: 'dict'

tasks 具有以下值：

[{'category': 'aaa',
  'task': <coroutine object get_docker_list at 0x000001B12B8560C0>},
 {'category': 'bbb',
  'task': <coroutine object get_docker_list at 0x000001B12B856F40>},
 {'category': 'ccc',
  'task': <coroutine object get_docker_list at 0x000001B12B856740>}]

顺便说一句，这是一种限制 http 提取调用的方法吗？例如，最多同时获取四个运行ning。

Answer 1

gather 接受协程（或其他可等待的）参数和 returns 它们结果的元组以相同的顺序。您正在向它传递一系列字典，其中一些值是协程。 gather 不知道该怎么做，并试图将字典视为可等待的对象，但很快就失败了。

生成字典列表的正确方法是只将协程传递给 gather，等待结果，然后将它们处理成新的字典：

async def get_all_details():
    category_list = await get_categories()
    details_list = await asyncio.gather(
        *[get_details(category) for category in category_list]
    )
    return [
        {'category': category, 'details': details}
        for (category, details) in zip(category_list, details_list)
    ]

BTW, is it a way to throttle the http fetch calls? For example, at most four fetches running at the same time.

限制并行调用的方便且惯用的方法是使用 semaphore:

async def get_details(category, limit):
    # acquiring the semaphore passed as `limit` will allow at most a
    # fixed number of coroutines to proceed concurrently
    async with limit:
        ... the rest of the code ...

async def get_all_details():
    limit = asyncio.Semaphore(4)
    category_list = await get_categories()
    details_list = await asyncio.gather(
        *[get_details(category, limit) for category in category_list]
    )
    ... the rest of the code ...

asyncio.gather() 在具有协程字段的字典列表上？

asyncio.gather() on list of dict which has a field of coroutine?

python

python-asyncio

tornado