为什么 "requests" 模块超时在 python3 中不起作用？

Question

我想获取 URL 的内容，如果我遇到 50X HTTP 错误代码，请重试 10 次，两次重试之间有 0.5 秒的延迟。我还想为我的请求设置 1 秒超时。为了实现这个目标，我尝试了以下程序：

#!/usr/bin/python3

import sys
import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter    

URL = "http://<DOMAIN>/Loader.aspx?ParTree=15131H&i=" # + stockid


def get_data(stockid):

    print("1")
    req_session = requests.Session()

    print("2")
    retries = Retry(total = 10, 
                    backoff_factor = 0.5,
                    status_forcelist = [500, 502, 503, 504])

    print("3")
    req_session.mount("http://", HTTPAdapter(max_retries=retries))

    print("4")
    page_content = req_session.get(URL + stockid, timeout=1).content

    print("5")

get_data(sys.argv[1])

不幸的是，它在屏幕上打印 4 后冻结了几分钟。怎么了？

Answer 1

首先，我是 Windows 用户。您的 timeout 工作正常。这里 backoff_factor 造成了问题。尝试：

#!/usr/bin/python3

import sys
import requests
from requests.packages.urllib3.util.retry import Retry # requests.packages. is not necessary
from requests.adapters import HTTPAdapter    

URL = "http://<DOMAIN>/Loader.aspx?ParTree=15131H&i=" # + stockid


def get_data(stockid):

    print("1")
    req_session = requests.Session()

    print("2")
    retries = Retry(total = 10, 
                    backoff_factor = 0.5,
                    status_forcelist = [500, 502, 503, 504])
    requests.packages.urllib3.util.Retry.BACKOFF_MAX = 0.5 # required
    # requests.packages. is not necessary
    # it isin't necessary you have to mount this way
    print("3")
    adapter = HTTPAdapter(max_retries=retries)
    req_session.mount('http://', adapter)
    req_session.mount('https://', adapter)

    print("4")
    page_content = req_session.get(URL + stockid, timeout=1).content

    print("5")

get_data(sys.argv[1])

这种方式也可以：

retries = RetryRequest(
        total=10,
        backoff_factor=0.5,
        status_forcelist=[500, 502, 503, 504],
        max_backoff=backoff_factor)

据记载，当您使用 backoff_factor:

It will never be longer than Retry.BACKOFF_MAX ( which is by default 120).

因此，当您尝试 link 时，BACKOFF_MAX 先于 backoff_factor，这会增加您的执行时间。因此，设置：

requests.packages.urllib3.util.Retry.BACKOFF_MAX = 0.5

会帮助你。
但不应引发此错误，因为它定义为：

min(self.BACKOFF_MAX, backoff_value)

在：

urllib3.util.retry

为什么 "requests" 模块超时在 python3 中不起作用？

Why "requests" module timeout doesn't work in python3?

urllib3

python-3.x

python-requests