urllib3：抛出 MaxRetryError 时如何获得响应？

Question

我正在使用 Python urllib3 + requests 的重试模块来处理第 3 方 API 偶尔出现错误的情况。我遇到的一个问题是，如果重试一直失败，我会得到 exceptions.MaxRetryError 并且永远看不到响应是什么。如果有来自服务器的有价值的调试数据怎么办？

有没有办法在抛出 MaxRetryError 的情况下仍然获得响应？

下面是我的代码

from requests.packages.urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

def req_with_retry(retries=3, backoff_factor=0.5, status_forcelist=(400, 404, 500, 502, 504,), method_whitelist=frozenset(['POST', 'HEAD', 'TRACE', 'GET', 'PUT', 'OPTIONS', 'DELETE']), session=None,):
    ''' 
    this returns a session that functions like the requests module but with retries built it for certain status codes
    '''
    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
        method_whitelist=method_whitelist
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session

r = req_with_retry().get(url="https://www.google.com")

Answer 1

首先，我想提一下，不再需要 from requests.packages.urllib3.util.retry import Retry，因为 urllib3 已从请求中取消供应。考虑改用 from urllib3.util.retry import Retry 形式，因为它不涉及黑客攻击（参见 https://github.com/psf/requests/blob/v2.22.0/requests/packages.py）。

您无法同时获得 MaxRetryError 和来自 urllib3 的响应。但是您可以从 urllib3 获得响应，并在请求级别使用 raise_for_status。但是，raise_for_status() 不会受到您的 status_forcelist 的影响，因此您可能需要自己重新实现它。

这是对您的代码的改编，它 1/ 重试状态 2/ 打印最终响应 3/ 引发异常。

import logging

import urllib3
import requests
import requests.adapters

logging.basicConfig(level=logging.DEBUG)
logging.getLogger("urllib3").setLevel(logging.DEBUG)


def req_with_retry(
    retries=3,
    backoff_factor=0.5,
    status_forcelist=(400, 404, 500, 502, 504),
    method_whitelist=frozenset(
        ["POST", "HEAD", "TRACE", "GET", "PUT", "OPTIONS", "DELETE"]
    ),
    session=None,
):
    """Returns a session that functions like the requests module but with retries
    built it for certain status codes
    """
    session = session or requests.Session()
    retry = urllib3.Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
        method_whitelist=method_whitelist,
        raise_on_status=False,
    )
    adapter = requests.adapters.HTTPAdapter(max_retries=retry)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    return session


r = req_with_retry().get(url="https://httpbin.org/status/404")
print(r.status_code, r.text)
r.raise_for_status()

urllib3：抛出 MaxRetryError 时如何获得响应？

urllib3: How to get response when MaxRetryError is thrown?

python

python-requests

urllib3