Scrapy Splash ERROR: Gave up retrying 504 Gateway Time-out

Question

我在使用 splash 和 scrapy 时收到此 504 网关错误，同时在我尝试抓取此内容的地方学习 splash https://www.lazada.com.my/

你能帮帮我吗？

Splash 运行在端口 8050 上的 docker 容器上

蜘蛛文件

import scrapy
from scrapy_splash import SplashRequest

class LaptopSpider(scrapy.Spider):
    name = 'laptop'
    allowed_domains = ['www.lazada.com.my']

    def start_requests(self):
        url='https://www.lazada.com.my/shop-laptops/?spm=a2o4k.home.cate_2.2.75f82e7eO7Jbgl'
        yield SplashRequest(url=url)

    def parse(self, response):
        all_rows=response.xpath("//div[@class='_17mcb']/div").getall()
        print(all_rows)
        for row in all_rows:
            title=row.xpath(".//div/div/div[2]/div[2]/a/text()")
            yield{
                'title':title
            }

设置

BOT_NAME = 'lazada'
SPIDER_MODULES = ['lazada.spiders']
NEWSPIDER_MODULE = 'lazada.spiders'
ROBOTSTXT_OBEY = False
SPLASH_URL = 'http://localhost:8050'
DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

Answer 1

您尝试抓取的 url 需要很长时间才能加载。即使您在浏览器中尝试，您也会注意到完全加载和停止旋转需要时间。

Splash 因此在页面完全加载并返回之前超时。

你需要做两件事。

首先增加启动启动服务器时的最大超时值，如下所示。

docker run -p 8050:8050 scrapinghub/splash --max-timeout 3600

其次，在蜘蛛中，您可以提供一个小于或等于启动服务器的max-timeout值的超时值。

yield SplashRequest(url=url, args={"timeout": 3000})

Scrapy Splash ERROR: Gave up retrying 504 Gateway Time-out

Scrapy Splash ERROR: Gave up retrying 504 Gateway Time-out

python

web-scraping

lua

scrapy

scrapy-splash