如何在处理请求重试时覆盖 BACKOFF_MAX

Question

我正在使用带有 python3 的请求模块来实现失败时的重试机制。

以下是我的代码-

session = session or requests.Session()
retry = Retry(
    total=retries,
    read=retries,
    connect=retries,
    backoff_factor=backoff_factor,
    status_forcelist=status_forcelist,
    method_whitelist=method_whitelist)
retry.BACKOFF_MAX = 60
adapter = HTTPAdapter(max_retries=retry)
session.mount(BASE_URL, adapter)
return session

我没有看到任何命名参数来设置最大退避（重试 class 中的 BACKOFF_MAX）。我不希望重试之间的睡眠时间超过 60 秒。我怎样才能实现它？重置 BACKOFF_MAX 无效

Answer 1

我通过重写 get_backoff_time() 的 Retry class

使它工作

class RetryRequest(Retry):

    DEFAULT_METHOD_WHITELIST = frozenset([
    'HEAD', 'GET', 'PUT', 'DELETE', 'OPTIONS', 'TRACE'])

    def __init__(self, total=10, connect=None, read=None, redirect=None, status=None,
             method_whitelist=DEFAULT_METHOD_WHITELIST, status_forcelist=None,
             backoff_factor=0, raise_on_redirect=True, raise_on_status=True,
             history=None, respect_retry_after_header=True, max_backoff=60):
        super(RetryRequest, self).__init__(total=total, connect=connect, read=read, redirect=redirect, status=status,
             method_whitelist=method_whitelist, status_forcelist=status_forcelist,
             backoff_factor=backoff_factor, raise_on_redirect=raise_on_redirect, raise_on_status=raise_on_status,
             history=history, respect_retry_after_header=respect_retry_after_header)
        self.backoff_max = max_backoff

    def get_backoff_time(self):
    """ Formula for computing the current backoff
    :rtype: float
    """
        # We want to consider only the last consecutive errors sequence (Ignore redirects).
        consecutive_errors_len = len(list(takewhile(lambda x: x.redirect_location is None,
                                                reversed(self.history))))
        if consecutive_errors_len <= 1:
            return 0

        backoff_value = self.backoff_factor * (2 ** (consecutive_errors_len - 1))
        return min(self.backoff_max, backoff_value)

并使用此 class 而不是 Retry

session = session or requests.Session()
retry = RetryRequest(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
        method_whitelist=method_whitelist,
        max_backoff=max_backoff # in my case 60
)
adapter = HTTPAdapter(max_retries=retry)
session.mount(BASE_URL, adapter)
return session

Answer 2

我遇到了同样的问题，这对我有用：

class RetryRequest(Retry):
    def __init__(self, backoff_max=60, **kwargs):
    super(RetryRequest, self).__init__(**kwargs)
    self.BACKOFF_MAX = backoff_max

我认为您不需要覆盖 get_backoff_time()，只需覆盖构造函数并设置 self.BACKOFF_MAX 即可。此外，使用 kwargs 可以省略其他参数。

Answer 3

这是我实现我想要的行为的技巧:

from urllib3.util.retry import Retry

class RetryRequest(Retry):
    BACKOFF_MAX = 77

优点：

您不必覆盖 Retry 的任何方法 (urllib3.util.retry)

缺点：

不是真正的参数

其他答案似乎可以解决 "real parameter" 的情况。但是，您还必须覆盖 new() 方法：

from urllib3.util.retry import Retry

class RetryRequest(Retry):
    def __init__(self, backoff_max=Retry.BACKOFF_MAX, **kwargs):
        super().__init__(**kwargs)
        self.BACKOFF_MAX = backoff_max

    def new(self, **kwargs):
        return super().new(backoff_max=self.BACKOFF_MAX, **kwargs)

如果您不覆盖它，它将使用 __init__() 定义中设置的默认参数。

不要犹豫，用以下方法测试您的解决方案：

total=100
retry = RetryRequest(total=total, backoff_factor=0.1, backoff_max=77)

for i in range(0, total):
    print("Try #%d" % (total - retry.total + 1))
    retry = retry.increment()
    print(retry.get_backoff_time())

Answer 4

基于中的函数覆盖想法，如果您希望它在全球范围内应用，请使用更简单的版本：

orig_backoff_fun = Retry.get_backoff_time

def custom_backoff_time(self):
    return min(5, orig_backoff_fun(self))

Retry.get_backoff_time = custom_backoff_time

将 5 替换为您想要的最长等待时间。

如何在处理请求重试时覆盖 BACKOFF_MAX

How to override BACKOFF_MAX while working with requests retry

python

python-3.x

python-requests

urllib3