Scrapyd 重试,如果 Errback 在扭曲的 deferreds / 无法打开 CONNECT 隧道错误
Scrapyd Retry if Errback in twisted deferreds / Could not open CONNECT tunnel Error
我正在使用 Scrapy 中的代理循环,有时一些代理会出错,这会阻止我的蜘蛛..
错误是 "Could not open CONNECT tunnel"
如果出现此错误,我该如何更改代码以使用另一个代理重试?
这里是需要修改的代码:
https://github.com/scrapy/scrapy/blob/master/scrapy/core/downloader/handlers/http11.py
像这样:
from scrapy.core.downloader.handlers.http11 import TunnelError
class RetryMiddleware(RetryMiddleware):
def process_exception(self, request, exception, spider):
if ( isinstance(exception, self.EXCEPTIONS_TO_RETRY) or isinstance(exception, TunnelError) ) \
and 'dont_retry' not in request.meta:
return self._retry(request, exception, spider)
在settings.py中:
DOWNLOADER_MIDDLEWARES = {
'myproject.myretry.RetryMiddleware': 200,
'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': None
}
我正在使用 Scrapy 中的代理循环,有时一些代理会出错,这会阻止我的蜘蛛.. 错误是 "Could not open CONNECT tunnel" 如果出现此错误,我该如何更改代码以使用另一个代理重试?
这里是需要修改的代码: https://github.com/scrapy/scrapy/blob/master/scrapy/core/downloader/handlers/http11.py
像这样:
from scrapy.core.downloader.handlers.http11 import TunnelError
class RetryMiddleware(RetryMiddleware):
def process_exception(self, request, exception, spider):
if ( isinstance(exception, self.EXCEPTIONS_TO_RETRY) or isinstance(exception, TunnelError) ) \
and 'dont_retry' not in request.meta:
return self._retry(request, exception, spider)
在settings.py中:
DOWNLOADER_MIDDLEWARES = {
'myproject.myretry.RetryMiddleware': 200,
'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': None
}