How to resolve OSError: [Errno 24] Too many open files when performing POST calls with three different URLs using aiohttp?

Question

我有一个 csv 文件，其中一列用作在两个 POST 方法调用中传递的数据。我有三个不同的 URLs 需要在代码中随后调用。我使用 aiohttp 定义会话并循环列长度（其中一个单元格值在两个 POST 调用中发送）。从响应中获得的数据存储在一个列表中，用于第三次 POST 不同 url 的调用。

前两个调用似乎执行得很好而且相当快（我为它们打印了响应并进行了验证），但是当执行到第三个时，出现了这个错误：

OSError: [Errno 24] Too many open files

我尝试了一些建议在 aiohttp.ClientSession(connector=aiohttp.TCPConnector(verify_ssl=False)) 中指定 connector=aiohttp.TCPConnector(verify_ssl=False) 的解决方案但这不起作用。解决这个问题的最佳方法应该是什么？我目前使用的第二个 URL 是 HTTP 本地主机 url，最终将更改为基于 https 云 url。

下面是描述情况的示例代码：

import aiohttp
import pandas as pd
import asyncio
import requests
import json
import time

start_time = time.time()

df = pd.read_csv('Sample.csv', header=None, dtype=str)
RequestBodyvalues = df.iloc[:, [0]].values

async def get_FirstUrlresponse(session, url, requestBody, headers):
    async with session.post(url, data = json.dumps(requestBody), headers = headers) as resp:
        response = await resp.json()
        return response

async def get_SecondUrlresponse(session, url, requestBody, headers):
    async with session.post(url, data = json.dumps(requestBody), headers = headers) as resp:
        response = await resp.json()
        return response

async def get_ThirdUrlresponse(session, url, requestBody, headers):
    async with session.post(url, data = json.dumps(requestBody), headers = headers) as resp:
        response = await resp.json()
        return response

async def main():
    async with aiohttp.ClientSession() as session:
        FirstUrlTasks = []
        SecondUrlTasks = []
        for reqBody in RequestBodyvalues:
            firstUrl = 'https://firstUrl.com/searchByValue'
            secondUrl = 'http://secondUrl.com/someEndpoint'
            requestBody = {'value': reqBody}
            headers = {'Authorization' : 'Bearer Token', 
            'content-type': 'application/json'}

            FirstUrlTasks.append(asyncio.ensure_future(get_FirstUrlresponse(session, firstUrl, requestBody, headers)))
            SecondUrlTasks.append(asyncio.ensure_future(get_SecondUrlresponse(session, secondUrl, requestBody, headers)))
        
        firstUrlResponses = await asyncio.gather(*FirstUrlTasks)
        secondUrlresponses = await asyncio.gather(*SecondUrlTasks)

        valuesForThridUrl = []
        for secondUrlresponse in secondUrlresponses:
            #Logic to fetch values to pass to Third Url stored in list

        ThirdUrlTasks = []
        for value in valuesForThridUrl:
            ThirdUrl = 'https://thirdUrl.com/someEndpoint'
            requestBody = {'reqBody': value}
            headers = {'Authorization' : 'Bearer Token', 
            'content-type': 'application/json'}

            ThirdUrlTasks.append(asyncio.ensure_future(get_ThirdUrlresponse(session, ThirdUrl, requestBody, headers)))

        thirdUrlresponses = await asyncio.gather(*ThirdUrlTasks)

asyncio.run(main())

Answer 1

使用此命令查看打开的文件限制

ulimit -n

然后增加限制

ulimit -n NUM

How to resolve OSError: [Errno 24] Too many open files when performing POST calls with three different URLs using aiohttp?

How to resolve OSError: [Errno 24] Too many open files when performing POST calls with three different URLs using aiohttp?

python

asynchronous

python-asyncio

aiohttp