Scrapy：抓取非常 select 个 URL

Question

我正试图为一个学校项目收集雅虎股票，但我不知道如何以非常确定的 link 浏览页面的每个 link。目标是用 url 的特定结尾部分遍历每只股票，如下所示：

Starting URL = ["https://ca.finance.yahoo.com/q/hp?s=BMO.TO&a=02&b=2&c=2005&d=02&e=2&f=2015&g=m"]

下一个 URL 会是这样的：

#Canadian Imperial(note the "CM"):
"https://ca.finance.yahoo.com/q/hp?s=CM.TO&a=02&b=2&c=2005&d=02&e=2&f=2015&g=m"

#Blackberry (note the "BB"):
"https://ca.finance.yahoo.com/q/hp?s=BB.TO&a=02&b=2&c=2005&d=02&e=2&f=2015&g=m"

等...

换句话说，唯一会改变的是 "hp?s=" 和 ".TO&a"[=35= 之间的字符].

想知道这是否可能。 URL 的结尾部分必须与我需要到达的页面保持一致。不幸的是，yahoo 上的每个页面中都没有 link 可以转到其他股票。

如果我可以用 Scrapy 的规则和 SmglLinkExtractor 来做到这一点，那就更好了。

非常感谢任何帮助！

谢谢！

当前的 Scrapy 代码：

from scrapy.spider import Spider from scrapy.selector import Selector from dirbot.items import Website from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.contrib.linkextractors import LinkExtractor class DmozSpider(Spider): name = "dmoz" allowed_domains = ["ca.finance.yahoo.com"] start_urls = [ "https://ca.finance.yahoo.com/q/hp?s=BMO.TO&a=02&b=2&c=2005&d=02&e=2&f=2015&g=m" ] rules = [ Rule(LinkExtractor(allow=r"/q/hp\?s=\w+\.TO&a=02&b=2&c=2005&d=02&e=2&f=2015&g=m"), follow=True) ] def parse(self, response): item = Website() item['name'] = response.xpath('//div[@class="title"]/h2/text()').extract() print item['name']

Answer 1

如果您有要为其加载雅虎页面的股票列表，您可以获得这样的雅虎网址列表：

url_template = "https://ca.finance.yahoo.com/q/hp?s={}.TO&a=02&b=2&c=2005&d=02&e=2&f=2015&g=m"

stocks = ['CM', 'BB']
urls = [url_template.format(stock) for stock in stocks]

虽然我没有使用过 scrapy，所以我不确定这是否是您需要的。

Answer 2

这是我在留下的评论中谈论的示例。

import urllib
import os

company_symbol = ["ACGL", "AFSI", "AGII", "AGNC", "ANAT", "ARCP", "ASBC", "ASPS", "BANF", "BBCN", "BGCP", "BNCL", "BOKF", "BPOP", "BRKL", "CACC", "CATY", "CBOE", "CBSH", "CFFN", "CHFC", "CINF", "CME ", "COLB", "CVBF", "ERIE", "ESGR", "ETFC", "EWBC", "EZPW", "FCFS", "FCNC", "FFBC", "FFIN", "FITB", "FMBI", "FMER", "FNFG", "FNGN", "FSRV", "FULT", "GBCI", "GLPI", "GLRE", "HBAN", "HBHC", "HLSS", "HOMB", "IBKC", "IBKR", "IBOC", "IPCC", "ISBC", "KRNY", "LPLA", "MBFI", "MHLD", "MKTX", "MTGE", "NAVG", "NBTB", "NDAQ", "NFBK", "NPBC", "NTRS", "NWBI", "ORIT", "OZRK", "PACW", "PBCT", "PCH ", "PNFP", "PRAA", "PVTB", "ROIC", "SAFT", "SBNY", "SBRA", "SCBT", "SEIC", "SIGI", "SIVB", "SLM ", "STFC", "SUSQ", "TCBI", "TFSL", "TRMK", "TROW", "UBSI", "UMBF", "UMPQ", "VRTS", "WABC", "WAFD", "WETF", "WRLD", "WTFC", "Z", "ZION"]

for company in company_symbol:
    url = 'http://finance.google.com/finance/info?client=ig&q={0}:{1}'.format(company, 'NASDAQ')
    nasdaq = urllib.urlopen(url)
    text = nasdaq.read()
    filename = 'nasdaq.txt'.format(company)
    with file(filename, 'a') as output:
        output.write(str(text))

此代码将作为更改 urls 并对每个 url 做某事的一种方法的示例编写。

Answer 3

为 follow 匹配模式的链接制定规则：

rules = [
    Rule(LinkExtractor(allow=r"/q/hp\?s=\w+\.\w+&a=\d+&b=\d+&c=\d+&d=\d+&e=\d+&f=\d+&g=m"), follow=True)
]

不过，我不确定您是否需要在此处检查所有 URL 参数。简化版：

rules = [
    Rule(LinkExtractor(allow=r"/q/hp\?s=\w+\.\w+"), follow=True)
]

还有，不要忘记导入：

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor

Answer 4

如果您只需要抓取给定时间段内的预定义报价，则逻辑如下：

准备您感兴趣的报价列表，例如['ABC'、'XYZ'、'LOL'、...]。
使用基本 scrapy.Spider.
定义 start_requests() 方法并从中产生一系列请求。

实施示例：

# -*- coding: utf-8 -*-
import scrapy
from scrapy.http import Request


class QuotesSpider(scrapy.Spider):

    name = "quotes"
    allowed_domains = ["ca.finance.yahoo.com"]
    quotes = ["BMO", "CM", "BB"]
    url_template = "https://ca.finance.yahoo.com/q/hp?s=%s.TO\
        &a=02&b=2&c=2005&d=02&e=2&f=2015&g=m"

    def start_requests(self):
        for quote in self.quotes:
            url = self.url_template % quote
            yield Request(url)

    def parse(self, response):
        # process

但如果您需要获取所有 TSX 报价数据，那么我建议您从 available listings 中抓取它们，然后像上面的示例一样使用。抓取整个 ca.finance.yahoo.com 显然不是一个好主意。

Scrapy：抓取非常 select 个 URL

Scrapy: Scraping very select URLs

python

scrapy

当前的 Scrapy 代码：