非常简单的 scrapy 爬虫不跟踪链接

Question

这是一个简单的 scrapy 爬虫，它可以抓取 yelp.com 并获取数据

我设置了Rule(LinkExtractor(allow=('.*')),follow=True,callback="parseBusiness")

跟踪链接和回调作为 parseBusiness

但是，这里的 Scrapy 不会跟随链接

这是具体的输出（这里是完整输出http://pastebin.com/BkuErvMq）

2015-07-14 01:06:22 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-07-14 01:06:25 [scrapy] DEBUG: Crawled (200) <GET http://www.yelp.com/search?find_desc=Hotels&find_loc=San+Francisco%2C+CA&ns=1> (referer: None)
2015-07-14 01:06:26 [scrapy] DEBUG: Crawled (200) <GET http://www.yelp.com/biz/ucsf-medical-center-at-mount-zion-san-francisco> (referer: None)
2015-07-14 01:06:26 [scrapy] INFO: Closing spider (finished)
2015-07-14 01:06:26 [scrapy] INFO: Dumping Scrapy stats:

这是我下面的代码

import sys
import scrapy

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor


class Business(scrapy.Item):
    name = scrapy.Field()
    contactNumber = scrapy.Field()
    address = scrapy.Field()

class YelpSpider(CrawlSpider):
    name = "yelp"
    allowed_domains = ["www.yelp.com"]
    start_urls = [
    "http://www.yelp.com/search?find_desc=Hotels&find_loc=San+Francisco%2C+CA&ns=1",
    "http://www.yelp.com/biz/ucsf-medical-center-at-mount-zion-san-francisco"
]

Rule(LinkExtractor(allow=()),follow=True,callback="parseBusiness")

def parseBusiness(self, response):
        business = Business()
        business['name'] = stripchars(response.xpath('//h1[@itemprop="name"]//text()').extract())
        business['contactNumber'] = stripchars(response.xpath('//span[@itemprop="telephone"]//text()').extract())
        business['address'] = stripchars(response.xpath('//li[@class="address"]//text()').extract())
        yield business

我在这里错过了什么？让 scrapy 跟踪所有链接

Answer 1

您没有设置蜘蛛的 rules 属性：

class YelpSpider(CrawlSpider):
    name = "yelp"
    allowed_domains = ["www.yelp.com"]
    start_urls = [
        "http://www.yelp.com/search?find_desc=Hotels&find_loc=San+Francisco%2C+CA&ns=1",
        "http://www.yelp.com/biz/ucsf-medical-center-at-mount-zion-san-francisco"
    ]

    rules = [
        Rule(LinkExtractor(allow=('.*')),follow=True,callback="parseBusiness")
    ]

非常简单的 scrapy 爬虫不跟踪链接

very simple scrapy crawler not following links

python

scrapy

scrapy-spider