Scrapy 选择器不适用于 Splash 响应
Scrapy selector not working on Splash response
我正在尝试使用 Scrapy 抓取一些动态内容。
我已经成功地设置了 Splash 来配合它工作。
但是，以下蜘蛛的选择器产生空结果：
# -*- coding: utf-8 -*-
import scrapy
from scrapy.selector import Selector
from scrapy_splash import SplashRequest
class CartierSpider(scrapy.Spider):
name = 'cartier'
start_urls = ['http://www.cartier.co.uk/en-gb/collections/watches/mens-watches/ballon-bleu-de-cartier/w69017z4-ballon-bleu-de-cartier-watch.html']
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, self.parse, args={'wait': 0.5})
def parse(self, response):
yield {
'title': response.xpath('//title').extract(),
'link': response.url,
'productID': Selector(text=response.body).xpath('//span[@itemprop="productID"]/text()').extract(),
'model': Selector(text=response.body).xpath('//span[@itemprop="model"]/text()').extract(),
'price': Selector(text=response.body).css('div.price-wrapper').xpath('.//span[@itemprop="price"]/text()').extract(),
}
选择器使用 Scrapy 工作得很好 shell，所以我很困惑什么不起作用。
我能在这两种情况中找到的唯一区别是字符串 response.body 的编码处理方式不同：如果我尝试从 [=12] 中 print/decode 它只是乱码=]函数。
任何提示或参考将不胜感激。
我试过 SplashRequest 和 运行 遇到与您相同的问题。乱七八糟之后，我尝试改为执行 LUA 脚本。
script = """
function main(splash)
local url = splash.args.url
assert(splash:go(url))
assert(splash:wait(0.5))
return {
html = splash:html(),
png = splash:png(),
har = splash:har(),
}
end
"""
然后使用脚本作为参数发出请求。您可以乱用脚本。在 localhost:9200 的 shell 或您选择的其他端口上进行测试。
yield SplashRequest(
url,
self.parse, args={'lua_source': self.script}, endpoint='execute')
哦，顺便说一句，您提供信息的方式很奇怪，请改用项目。
你的蜘蛛对我来说工作得很好，使用 Scrapy 1.1，Splash 2.1 并且没有修改你问题中的代码，只是使用 https://github.com/scrapy-plugins/scrapy-splash
中建议的设置
正如其他人所提到的，您的 parse 函数可以通过直接使用 response.css() 和 response.xpath() 来简化，而无需从响应中重新构建 Selector .
我试过：
import scrapy
from scrapy.selector import Selector
from scrapy_splash import SplashRequest
class CartierSpider(scrapy.Spider):
name = 'cartier'
start_urls = ['http://www.cartier.co.uk/en-gb/collections/watches/mens-watches/ballon-bleu-de-cartier/w69017z4-ballon-bleu-de-cartier-watch.html']
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, self.parse, args={'wait': 0.5})
def parse(self, response):
yield {
'title': response.xpath('//title/text()').extract_first(),
'link': response.url,
'productID': response.xpath('//span[@itemprop="productID"]/text()').extract_first(),
'model': response.xpath('//span[@itemprop="model"]/text()').extract_first(),
'price': response.css('div.price-wrapper').xpath('.//span[@itemprop="price"]/text()').extract_first(),
}
得到这个：
$ scrapy crawl cartier
2016-06-08 17:16:08 [scrapy] INFO: Scrapy 1.1.0 started (bot: stack37701774)
2016-06-08 17:16:08 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'stack37701774.spiders', 'SPIDER_MODULES': ['stack37701774.spiders'], 'BOT_NAME': 'stack37701774'}
(...)
2016-06-08 17:16:08 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy_splash.SplashCookiesMiddleware',
'scrapy_splash.SplashMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-06-08 17:16:08 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-06-08 17:16:08 [scrapy] INFO: Enabled item pipelines:
[]
2016-06-08 17:16:08 [scrapy] INFO: Spider opened
2016-06-08 17:16:08 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-06-08 17:16:08 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-06-08 17:16:11 [scrapy] DEBUG: Crawled (200) <GET http://www.cartier.co.uk/en-gb/collections/watches/mens-watches/ballon-bleu-de-cartier/w69017z4-ballon-bleu-de-cartier-watch.html via http://localhost:8050/render.html> (referer: None)
2016-06-08 17:16:11 [scrapy] DEBUG: Scraped from <200 http://www.cartier.co.uk/en-gb/collections/watches/mens-watches/ballon-bleu-de-cartier/w69017z4-ballon-bleu-de-cartier-watch.html>
{'model': u'Ballon Bleu de Cartier watch', 'productID': u'W69017Z4', 'link': 'http://www.cartier.co.uk/en-gb/collections/watches/mens-watches/ballon-bleu-de-cartier/w69017z4-ballon-bleu-de-cartier-watch.html', 'price': None, 'title': u'CRW69017Z4 - Ballon Bleu de Cartier watch - 36 mm, steel, leather - Cartier'}
2016-06-08 17:16:11 [scrapy] INFO: Closing spider (finished)
2016-06-08 17:16:11 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 618,
'downloader/request_count': 1,
'downloader/request_method_count/POST': 1,
'downloader/response_bytes': 213006,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2016, 6, 8, 15, 16, 11, 201281),
'item_scraped_count': 1,
'log_count/DEBUG': 3,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 2,
'scheduler/dequeued/memory': 2,
'scheduler/enqueued': 2,
'scheduler/enqueued/memory': 2,
'splash/render.html/request_count': 1,
'splash/render.html/response_count/200': 1,
'start_time': datetime.datetime(2016, 6, 8, 15, 16, 8, 545105)}
2016-06-08 17:16:11 [scrapy] INFO: Spider closed (finished)
我没有足够的声誉来添加评论，所以我必须在这里做出回答。
如果我为 SplashRequest 设置 'Accept-Encoding': 'gzip'，则 Splash 2.1 返回 "malformed"（未压缩的 gziped 实际上是 speeking）html 内容时我遇到了类似的问题。
最后，我在这里找到了解决方案：https://github.com/scrapinghub/splash/pull/102
将 'Accept-Encoding': 'gzip' 更改为：
'Accept-Encoding': 'deflate'
我不知道为什么，但它有效。
我正在尝试使用 Scrapy 抓取一些动态内容。 我已经成功地设置了 Splash 来配合它工作。 但是，以下蜘蛛的选择器产生空结果：
# -*- coding: utf-8 -*-
import scrapy
from scrapy.selector import Selector
from scrapy_splash import SplashRequest
class CartierSpider(scrapy.Spider):
name = 'cartier'
start_urls = ['http://www.cartier.co.uk/en-gb/collections/watches/mens-watches/ballon-bleu-de-cartier/w69017z4-ballon-bleu-de-cartier-watch.html']
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, self.parse, args={'wait': 0.5})
def parse(self, response):
yield {
'title': response.xpath('//title').extract(),
'link': response.url,
'productID': Selector(text=response.body).xpath('//span[@itemprop="productID"]/text()').extract(),
'model': Selector(text=response.body).xpath('//span[@itemprop="model"]/text()').extract(),
'price': Selector(text=response.body).css('div.price-wrapper').xpath('.//span[@itemprop="price"]/text()').extract(),
}
选择器使用 Scrapy 工作得很好 shell，所以我很困惑什么不起作用。
我能在这两种情况中找到的唯一区别是字符串 response.body 的编码处理方式不同：如果我尝试从 [=12] 中 print/decode 它只是乱码=]函数。
任何提示或参考将不胜感激。
我试过 SplashRequest 和 运行 遇到与您相同的问题。乱七八糟之后，我尝试改为执行 LUA 脚本。
script = """
function main(splash)
local url = splash.args.url
assert(splash:go(url))
assert(splash:wait(0.5))
return {
html = splash:html(),
png = splash:png(),
har = splash:har(),
}
end
"""
然后使用脚本作为参数发出请求。您可以乱用脚本。在 localhost:9200 的 shell 或您选择的其他端口上进行测试。
yield SplashRequest(
url,
self.parse, args={'lua_source': self.script}, endpoint='execute')
哦，顺便说一句，您提供信息的方式很奇怪，请改用项目。
你的蜘蛛对我来说工作得很好，使用 Scrapy 1.1，Splash 2.1 并且没有修改你问题中的代码，只是使用 https://github.com/scrapy-plugins/scrapy-splash
中建议的设置正如其他人所提到的，您的 parse 函数可以通过直接使用 response.css() 和 response.xpath() 来简化，而无需从响应中重新构建 Selector .
我试过：
import scrapy
from scrapy.selector import Selector
from scrapy_splash import SplashRequest
class CartierSpider(scrapy.Spider):
name = 'cartier'
start_urls = ['http://www.cartier.co.uk/en-gb/collections/watches/mens-watches/ballon-bleu-de-cartier/w69017z4-ballon-bleu-de-cartier-watch.html']
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, self.parse, args={'wait': 0.5})
def parse(self, response):
yield {
'title': response.xpath('//title/text()').extract_first(),
'link': response.url,
'productID': response.xpath('//span[@itemprop="productID"]/text()').extract_first(),
'model': response.xpath('//span[@itemprop="model"]/text()').extract_first(),
'price': response.css('div.price-wrapper').xpath('.//span[@itemprop="price"]/text()').extract_first(),
}
得到这个：
$ scrapy crawl cartier
2016-06-08 17:16:08 [scrapy] INFO: Scrapy 1.1.0 started (bot: stack37701774)
2016-06-08 17:16:08 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'stack37701774.spiders', 'SPIDER_MODULES': ['stack37701774.spiders'], 'BOT_NAME': 'stack37701774'}
(...)
2016-06-08 17:16:08 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy_splash.SplashCookiesMiddleware',
'scrapy_splash.SplashMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-06-08 17:16:08 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-06-08 17:16:08 [scrapy] INFO: Enabled item pipelines:
[]
2016-06-08 17:16:08 [scrapy] INFO: Spider opened
2016-06-08 17:16:08 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-06-08 17:16:08 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-06-08 17:16:11 [scrapy] DEBUG: Crawled (200) <GET http://www.cartier.co.uk/en-gb/collections/watches/mens-watches/ballon-bleu-de-cartier/w69017z4-ballon-bleu-de-cartier-watch.html via http://localhost:8050/render.html> (referer: None)
2016-06-08 17:16:11 [scrapy] DEBUG: Scraped from <200 http://www.cartier.co.uk/en-gb/collections/watches/mens-watches/ballon-bleu-de-cartier/w69017z4-ballon-bleu-de-cartier-watch.html>
{'model': u'Ballon Bleu de Cartier watch', 'productID': u'W69017Z4', 'link': 'http://www.cartier.co.uk/en-gb/collections/watches/mens-watches/ballon-bleu-de-cartier/w69017z4-ballon-bleu-de-cartier-watch.html', 'price': None, 'title': u'CRW69017Z4 - Ballon Bleu de Cartier watch - 36 mm, steel, leather - Cartier'}
2016-06-08 17:16:11 [scrapy] INFO: Closing spider (finished)
2016-06-08 17:16:11 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 618,
'downloader/request_count': 1,
'downloader/request_method_count/POST': 1,
'downloader/response_bytes': 213006,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2016, 6, 8, 15, 16, 11, 201281),
'item_scraped_count': 1,
'log_count/DEBUG': 3,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 2,
'scheduler/dequeued/memory': 2,
'scheduler/enqueued': 2,
'scheduler/enqueued/memory': 2,
'splash/render.html/request_count': 1,
'splash/render.html/response_count/200': 1,
'start_time': datetime.datetime(2016, 6, 8, 15, 16, 8, 545105)}
2016-06-08 17:16:11 [scrapy] INFO: Spider closed (finished)
我没有足够的声誉来添加评论，所以我必须在这里做出回答。
如果我为 SplashRequest 设置 'Accept-Encoding': 'gzip'，则 Splash 2.1 返回 "malformed"（未压缩的 gziped 实际上是 speeking）html 内容时我遇到了类似的问题。
最后，我在这里找到了解决方案：https://github.com/scrapinghub/splash/pull/102
将 'Accept-Encoding': 'gzip' 更改为：
'Accept-Encoding': 'deflate'
我不知道为什么，但它有效。