Scrapy

Question

我正在尝试抓取一些项目，如下所示：

def parse(self, response):

    item = GameItem()
    item['game_commentary'] = response.css('tr td:nth-child(2)[style*=vertical-align]::text').extract()
    item['game_movement'] = response.xpath("//tr/td[1][contains(@style,'vertical-align: top')]/text()").extract()

    yield item

我的问题是我不想 yield 当前 response.xpath 或 response.css 选择器提取的所有项目。

有没有办法，在将这些命令分配给 item['game_commentary'] 和 item['game_movement'] 之前，应用 regex 或其他东西来过滤不希望产生的不需要的值？

Answer 1

我会研究 Item Loaders 来完成这个。您必须按如下方式重写您的解析：

def parse(self, response):
    loader = GameItemLoader(item=GameItem(), response=response)
    loader.add_css('game_commentary', 'tr td:nth-child(2)[style*=vertical-align]::text')
    loader.add_xpath('game_movement', "//tr/td[1][contains(@style,'vertical-align: top')]/text()")
    item = loader.load_item()
    yield item

您的 items.py 看起来像这样：

from scrapy.item import Item, Field
from scrapy.loader import ItemLoader
from scrapy.loader.processors import TakeFirst

class GameItemLoader(Item):
    # default input & output processors
    # will be executed for each item loaded,
    # except if a specific in or output processor is specified
    default_output_processor = TakeFirst()

    # you can specify specific input & output processors per field
    game_commentary_in = '...'
    game_commentary_out = '...'

class GameItem(RetviewsItem):
    game_commentary = Field()
    game_movement = Field()

Scrapy - 过滤产生的项目

Scrapy - filtering yielded items

xpath

css-selectors

xml-parsing

python-3.x