Scrapy 使用 SgmlLinkExtractor

Question

我正在尝试抓取表单的页面 http://www.wynk.in/music/song/variable_underscored_alphanumeric_string.html。我想从笔记本电脑上访问此类 URL，但由于这些 URL 仅适用于应用程序和 WAP，因此我将用户代理指定为 'Mozilla/5.0 (Linux; U; Android 2.3.4; fr-fr; HTC Desire Build/GRJ22) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1' 在 settings.py 中。我的代码文件显示为

from scrapy import Selector
from wynks.items import WynksItem

from scrapy.contrib.spiders import CrawlSpider, Rule

from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor


class MySpider(CrawlSpider):

name = "wynk"
#allowed_domains = ["wynk.in"]
start_urls = ["http://www.wynk.in/", ]
#start_urls = []
rules = (Rule(SgmlLinkExtractor(allow=[r'/music/song/\w+.html']), callback='parse_item', follow=True),)

def parse_item(self, response):
    hxs = Selector(response)
    if hxs:
        tds = hxs.xpath("//div[@class='songDetails']//tr//td")
        if tds:
            for td in tds.xpath('.//div'):
                titles = td.xpath("a/text()").extract()
                if titles:
                    for title in titles:
                        print title

我通过运行开始代码 scrapy crawl wynk -o abcd.csv -t csv

然而，我只得到这个结果已抓取 (200) http://www.wynk.in/> (referer: None) 2015-03-23 11:06:04+0530 [wynk] 信息：关闭蜘蛛（完成）我做错了什么？

Answer 1

由于主页上没有直接 link 到上述 URL，解决方法是获取所有 link，并通过创建递归请求递归访问 music/song 页面.将继承更改为继承自 Spider 而不是 CrawlSpider

Scrapy 使用 SgmlLinkExtractor

Scrapy using SgmlLinkExtractor

python

regex

scrapy