让 scrapy spider 爬取整个网站

Question

我正在使用 scrapy 来抓取我拥有的旧网站，我正在使用下面的代码作为我的蜘蛛。我不介意为每个网页输出文件，或者包含其中所有内容的数据库。但我确实需要能够让蜘蛛爬行整个事情，而我不必输入我目前必须做的每一个 url

import scrapy

class DmozSpider(scrapy.Spider):
    name = "dmoz"
    allowed_domains = ["www.example.com"]
    start_urls = [
        "http://www.example.com/contactus"
    ]

    def parse(self, response):
        filename = response.url.split("/")[-2] + '.html'
        with open(filename, 'wb') as f:
            f.write(response.body)

Answer 1

要抓取整个网站，您应该使用 CrawlSpider 而不是 scrapy.Spider

Here's an example

为了您的目的尝试使用这样的东西：

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class MySpider(CrawlSpider):
    name = 'example.com'
    allowed_domains = ['example.com']
    start_urls = ['http://www.example.com']

    rules = (
        Rule(LinkExtractor(), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        filename = response.url.split("/")[-2] + '.html'
        with open(filename, 'wb') as f:
            f.write(response.body)

另外，看看这个article

让 scrapy spider 爬取整个网站

Get scrapy spider to crawl entire site

python

scrapy

scrapy-spider