Scrapy spider错误处理(scrapy.core.scraper)

Question

阅读了几个小时的解决方案后，我仍然找不到问题的答案。我正在尝试抓取超市网页，我认为错误出在解析函数中。如果有人可以帮助我，请。

import scrapy
from bs4 import BeautifulSoup
import re
from fakesupermarketPrueba.items import fakesupermarketItem

class FakeSupermarketSpider(scrapy.Spider):
    name = 'FakeSupermarketSpider'
    allowed_domains = ['fakesupermarket.com.py']
    start_urls = ['http://www.fakesupermarket.com.py/category/10']

    all_link_categories = [
        'http://www.fakesupermarket.com.py/category/1218.aspx',
        'http://www.fakesupermarket.com.py/category/1231.aspx']

    def __init__(self):
        self.declare_xpath()


    def declare_xpath(self):

        self.TitleXpath = "/html/body/div[1]/form/div[5]/div[6]/div/div[2]/div[1]/div/div[1]/div[2]/div/h1/text()"
        self.LinkXpath = "/html/body/div[1]/form/@action"
        self.PriceXpath = "/html/body/div[1]/form/div[5]/div[6]/div/div[2]/div[1]/div/div[1]/div[2]/div/div[3]/div/div/div[5]/span/text()"
        self.NumSerieXpath = "/html/body/div[1]/form/div[5]/div[6]/div/div[2]/div[1]/div/div[1]/div[2]/div/div[3]/div/div/div[6]/text()"
        self.LinkImgXpath = "/html/body/div[1]/form/div[5]/div[6]/div/div[2]/div[1]/div/div[1]/div[1]/div[3]/ul/li/img/@src"
        self.UnitXpath = "/html/body/div[1]/form/div[5]/div[6]/div/div[2]/div[1]/div/div[1]/div[2]/div/div[3]/div/div/div[8]/div[1]/div/span/text()"


    def parse(self,response):
        for href in all_link_categories:
            url = response.urljoin(href)
            yield scrapy.Request(url=url, callback=self.parse_main_item)

    def parse_main_item(self,response):
        item = fakesupermarketItem()

        Title = response.xpath(self.TitleXpath).extract()
        Title = self.cleanText(self.parseText(self.listToStr(Title)))

        Link = response.xpath(self.LinkXpath).extract()
        Link = self.cleanText(self.parseText(Link))

        Price = response.xpath(self.PriceXpath).extract()
        Price = self.cleanText(self.parseText(self.listToStr(Price)))

        NumSerie = response.xpath(self.NumSerieXpath).extract()
        NumSerie = self.cleanText(self.parseText(self.listToStr(NumSerie)))

        LinkImg = response.xpath(self.LinkImgXpath).extract()
        LinkImg = self.cleanText(self.parseText(self.listToStr(LinkImg)))

        Unit = response.xpath(self.UnitXpath).extract()
        Unit = self.cleanText(self.parseText(Unit))

        item['Title']           = Title
        item['Link']            = Link
        item['Price']           = Price
        item['NumSerie']        = NumSerie
        item['LinkImg']         = LinkImg
        item['Unit']            = Unit
        return item

我的想法是复制最终产品所在的所有链接（大约有 300 个）。这些链接位于 all_link_categories 数组中。（我认为有错误）日志：

2021-01-12 20:14:40 [scrapy.core.engine] INFO: Spider opened
2021-01-12 20:14:40 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-01-12 20:14:40 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-01-12 20:14:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.fakesupermarket.com.py/robots.txt> (referer: None)
2021-01-12 20:14:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.fakesupermarket.com.py/category/10.aspx> (referer: None)
2021-01-12 20:14:42 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.fakesupermarket.com.py/category/10.aspx> (referer: None)
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/scrapy/utils/defer.py", line 120, in iter_errback
    yield next(it)
  File "/usr/local/lib/python3.8/dist-packages/scrapy/utils/python.py", line 353, in __next__
    return next(self.data)
  File "/usr/local/lib/python3.8/dist-packages/scrapy/utils/python.py", line 353, in __next__
    return next(self.data)
  File "/usr/local/lib/python3.8/dist-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
    for r in iterable:
  File "/usr/local/lib/python3.8/dist-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/usr/local/lib/python3.8/dist-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
    for r in iterable:
  File "/usr/local/lib/python3.8/dist-packages/scrapy/spidermiddlewares/referer.py", line 340, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/usr/local/lib/python3.8/dist-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
    for r in iterable:
  File "/usr/local/lib/python3.8/dist-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/usr/local/lib/python3.8/dist-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
    for r in iterable:
  File "/usr/local/lib/python3.8/dist-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/usr/local/lib/python3.8/dist-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
    for r in iterable:
  File "/root/fakesupermarketPrueba/fakesupermarketPrueba/spiders/spider.py", line 374, in parse
    for href in all_link_categories:
NameError: name 'all_link_categories' is not defined
2021-01-12 20:14:42 [scrapy.core.engine] INFO: Closing spider (finished)

Answer 1

为了访问 parse 方法
中蜘蛛定义中定义的 all_link_categories，您需要使用 self.all_link_categories 而不是 all_link_categories

...
        for href in self.all_link_categories:
            url = response.urljoin(href)
            yield scrapy.Request(url=url, callback=self.parse_main_item)

Scrapy spider错误处理(scrapy.core.scraper)

Scrapy spider error processing (scrapy.core.scraper)

python

scrapy

web-scraping

python-3.x

centos7