如何加速此 Python 循环

Question

    downloadStart = datetime.now()
while (True):
    requestURL = transactionAPI.format(page = tempPage,limit = 5000)
    response = requests.get(requestURL,headers=headers)
    json_data = json.loads(response.content)
    tempMomosTransactionHistory.extend(json_data["list"])  
    if(datetime.fromtimestamp(json_data["list"][-1]["crtime"]) <  datetime(datetime.today().year,datetime.today().month,datetime.today().day - dateRange)):          
        break                       
    tempPage += 1
downloadEnd = datetime.now()

有什么建议请穿线之类的吗？

这里输出

downloadtime 0:00:02.056010

downloadtime 0:00:05.680806

downloadtime 0:00:05.447945

Answer 1

你需要从两个方面改进它。

在循环内优化代码
并行代码执行

#1 通过查看您的代码，我可以看到一个改进，即。创建 datetime.today 对象而不是创建 3 次。检查其他方法，如 transactionAPI 进一步优化。

#2：如果你是多核 CPU 机器，那么你可以通过每页跨越线程来利用机器。参考上面修改后的代码。

import threading

def processRequest(tempPage):
    requestURL = transactionAPI.format(page = tempPage,limit = 5000)
    response = requests.get(requestURL,headers=headers)
    json_data = json.loads(response.content)
    tempMomosTransactionHistory.extend(json_data["list"])
    
downloadStart = datetime.now()
while (True):
     #create thread per page
     t1 = threading.Thread(target=processRequest, args=(tempPage, ))
     t1.start()
     #Fetch datetime today object once instaed 3 times
     datetimetoday = datetime()
    if(datetime.fromtimestamp(json_data["list"][-1]["crtime"]) <  datetime(datetimetoday.year,datetimetoday.month,datetimetoday.day - dateRange)):          
        break                       
    tempPage += 1
downloadEnd = datetime.now()

如何加速此 Python 循环

How to Speed Up This Python Loop

python

optimization

performance

multithreading