使用 asyncio 并发写入多个文件

Question

使用asyncio写入多个文件的设计模式是什么？

这是我想做的单线程版本的概要。


import datetime
import json

def generate_records():
   for i in range(100):
       # simulate some JSON content that I want to write to a file
       # assume that this takes some significant amount of time
       yield {
         'index':index,
         'message':f"Message {index}",
         'timestamp':datetime.datetime.now().isoformat()
       }



for (i,r) in enumerate(generate_records()):
    with open(f"record_{i:04}.json", 'w') as fout:
         json.dump(r, fout)

如何利用 asyncio 在生成记录的同时写入这些文件？

Answer 1

import json
import datetime
import aiofiles

async def main():
    def mk_record(index):
        # simulate some JSON content that I want to write to a file
        return {
            'index': index,
            'message': f"Message {index}",
            'timestamp': datetime.datetime.now().isoformat()
        }
    records = [mk_record(i) for i in range(100)]

    async def write_record(i, r):
        # Test by pretending the write takes a while
        # Delete this line for actual code.
        await asyncio.sleep(1)
        async with aiofiles.open(f'/tmp/record_{i:04}.json', mode='w') as fout:
            await fout.write(json.dumps(r))

    await asyncio.gather(*[write_record(i, r) for i, r in enumerate(records)])


if __name__ == '__main__':
    asyncio.run(main())

使用 asyncio 并发写入多个文件

Concurrent writing to multiple files using asyncio

python

python-asyncio