Aiohttp, Asyncio: RuntimeError: Event loop is closed

Question

我有两个脚本，scraper.py 和 db_control.py。在 scraper.py 我有这样的东西：

...
def scrape(category, field, pages, search, use_proxy, proxy_file):
    ...
    loop = asyncio.get_event_loop()

    to_do = [ get_pages(url, params, conngen) for url in urls ]
    wait_coro = asyncio.wait(to_do)
    res, _ = loop.run_until_complete(wait_coro)
    ...
    loop.close()
    
    return [ x.result() for x in res ]

...

在db_control.py中：

from scraper import scrape
...
while new < 15:
    data = scrape(category, field, pages, search, use_proxy, proxy_file)
    ...
...

理论上，抓取程序应该在未知时间启动，直到获得足够的数据。但是当 new 不是 imidiately > 15 时，就会发生这个错误：

  File "/usr/lib/python3.4/asyncio/base_events.py", line 293, in run_until_complete
self._check_closed()
  File "/usr/lib/python3.4/asyncio/base_events.py", line 265, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

但是如果我只运行 scrape() 一次，脚本就可以正常工作。所以我想重新创建 loop = asyncio.get_event_loop() 有一些问题，我试过 but nothing changed. How I can fix this? Of course those are just snippets of my code, if you think problem can be elsewhere, full code is available here.

Answer 1

方法run_until_complete, run_forever, run_in_executor, create_task, call_at明确检查循环并在关闭时抛出异常。

引自文档 - BaseEvenLoop.close:

This is idempotent and irreversible

除非你有一些（好的）理由，否则你可以简单地省略关闭线：

def scrape(category, field, pages, search, use_proxy, proxy_file):
    #...
    loop = asyncio.get_event_loop()

    to_do = [ get_pages(url, params, conngen) for url in urls ]
    wait_coro = asyncio.wait(to_do)
    res, _ = loop.run_until_complete(wait_coro)
    #...
    # loop.close()
    return [ x.result() for x in res ]

如果你想每次都有一个全新的循环，你不必手动创建它并设置为默认值：

def scrape(category, field, pages, search, use_proxy, proxy_file):
    #...
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)    
    to_do = [ get_pages(url, params, conngen) for url in urls ]
    wait_coro = asyncio.wait(to_do)
    res, _ = loop.run_until_complete(wait_coro)
    #...
    return [ x.result() for x in res ]

Aiohttp, Asyncio: RuntimeError: Event loop is closed

Aiohttp, Asyncio: RuntimeError: Event loop is closed

python

python-3.x

python-asyncio

aiohttp