GET 请求 Python 中的块

Question

我正在使用 requests 库从 API 获取数据，方法如下：

import requests

url = "Some String"

headers = {

'Authorization':"Some Token"}

response = requests.request("GET", url, headers=headers)

但是我尝试获取的文件非常大，所以我收到了时间异常错误。如何在请求中使用块来获取它？

谢谢！

Answer 1

import requests    
import datetime
import pandas as pd


url = "some URL"
headers = {

'Authorization':"Some Token"}

start_date = datetime.datetime(2018, 6, 1)
end_date = datetime.datetime.now()
temp_end_date = start_date + datetime.timedelta(days=7)

output = dict()
while temp_end_date <= end_date:
    temp_url = url % (start_date.timestamp()*1000, temp_end_date.timestamp()*1000)
    response = requests.get(temp_url, headers=headers)
    temp_data = response.json()
    for key, value in temp_data.items():
        output_arr = output.get(key, [])
        output_arr.extend(value)
        output[key] = output_arr
    start_date = temp_end_date + datetime.timedelta(seconds=1)
    temp_end_date += datetime.timedelta(days=7)

data=output
df=pd.DataFrame(data)

df.head()

GET 请求 Python 中的块

GET request with chunks in Python

python

chunks

python-requests