scrapy 中的下一步按钮

Question

我需要在 if 语句中使用 if 语句我已经必须确定我的抓取程序何时单击下一步按钮，以便我可以在发生这种情况时执行某些操作。当前的 if 语句只是判断页面上是否有下一个按钮。但我无法弄清楚如何确定何时实际单击下一个按钮。

            # Finds next page button
            priority = response.meta['priority']
            next_page = response.xpath('//a[contains(., "- Next>>")]/@href').get()
            # If it exists and there is a next page enter if statement
            if next_page is not None:
                # Go to next page
                yield response.follow(next_page, self.parse, priority=priority, meta={'priority': priority})

Answer 1

在 meta 键中有一个标志来确定 link 是否来自于单击 NEXT 按钮

def parse(self, response):

    if response.meta.get('isNextClicked', False):
        #Next was clicked

    # Finds next page button
    priority = response.meta['priority']    

    next_page = response.xpath('//a[contains(., "- Next>>")]/@href').get()
    # If it exists and there is a next page enter if statement
    if next_page is not None:
        # Go to next page
        yield response.follow(next_page, self.parse, priority=priority, meta={'priority': priority, 'isNextClicked': True})

scrapy 中的下一步按钮

Next button in scrapy

python

splash-screen

scrapy

web-scraping