抓取 0 页，抓取 0 项 ERROR / webscraping / SELENIUM
Crawled 0 pages, scraped 0 items ERROR / webscraping / SELENIUM
所以我尝试了几种方法来理解我的蜘蛛失败的原因，但没有成功。我已经被困了好几天了，不能再拖下去了。我只想抓取第一页，此时不进行分页。非常感谢您的帮助 :( 这是我的代码：
import scrapy
from scrapy_selenium import SeleniumRequest
class HomesSpider(scrapy.Spider):
name = 'homes'
def parse(self, response):
yield SeleniumRequest(
url='https://www.vivanuncios.com.mx/s-venta-inmuebles/queretaro/page-2/v1c1097l1021p2',
wait_time=3,
callback=self.parse
)
def parse(self, response):
homes = response.xpath("//div[@class='viewport-contents']/div")
for home in homes:
yield{
'price': home.xpath(".//span[@class='value wrapper']/span[@class='ad-price']/text()").get(),
'location': home.xpath(".//div[@class='tile-location one-liner']/b/text()").get(),
'description': home.xpath(".//div[@class='tile-desc one-liner']/a/text()").get(),
'bedrooms': home.xpath(".//div[@class='chiplets-inline-block re-bedroom']/text()").get(),
'm2': home.xpath(".//div[@class='chiplets-inline-block surface-area']/text()").get()
}
这是我的 settings.py 文件：
# Scrapy settings for real_state project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'real_state'
SPIDER_MODULES = ['real_state.spiders']
NEWSPIDER_MODULE = 'real_state.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'real_state (+http://www.yourdomain.com)'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
'User_Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36'
}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'real_state.middlewares.RealStateSpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
'scrapy_selenium.SeleniumMiddleware': 800
}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
# 'real_state.pipelines.RealStatePipeline': 300,
#}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
#SELENIUM
from shutil import which
SELENIUM_DRIVER_NAME = 'chrome'
SELENIUM_DRIVER_EXECUTABLE_PATH = which("C:\Users\Cesal\projects\real_state\chromedriver.exe")
SELENIUM_DRIVER_ARGUMENTS=['-headless'] # '--headless' if using chrome instead of firefox
这是我执行时在终端中得到的：
(base) PS C:\Users\Cesal\projects\real_state\real_state\spiders> scrapy crawl homes
2021-11-03 13:02:58 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: real_state)
2021-11-03 13:02:58 [scrapy.utils.log] INFO: Versions: lxml 4.6.1.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 21.2.0, Python 3.8.5 (default, Sep 3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1h 22 Sep 2020), cryptography 3.1.1, Platform Windows-10-10.0.19041-SP0
2021-11-03 13:02:58 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2021-11-03 13:02:58 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'real_state',
'NEWSPIDER_MODULE': 'real_state.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['real_state.spiders']}
2021-11-03 13:02:58 [scrapy.extensions.telnet] INFO: Telnet Password: ade49fc0492d5027
2021-11-03 13:02:58 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2021-11-03 13:02:59 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:64533/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "platformName": "any", "goog:chromeOptions": {"extensions": [], "args": ["-headless"]}}}, "desiredCapabilities": {"browserName": "chrome", "version": "", "platform": "ANY", "goog:chromeOptions": {"extensions": [], "args": ["-headless"]}}}
2021-11-03 13:02:59 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): 127.0.0.1:64533
DevTools listening on ws://127.0.0.1:64541/devtools/browser/302904be-ca13-4464-a332-8d995cb55f44
2021-11-03 13:03:00 [urllib3.connectionpool] DEBUG: http://127.0.0.1:64533 "POST /session HTTP/1.1" 200 788
2021-11-03 13:03:00 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2021-11-03 13:03:00 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy_selenium.SeleniumMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-11-03 13:03:00 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-11-03 13:03:00 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2021-11-03 13:03:00 [scrapy.core.engine] INFO: Spider opened
2021-11-03 13:03:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-11-03 13:03:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-11-03 13:03:00 [scrapy.core.engine] INFO: Closing spider (finished)
2021-11-03 13:03:00 [selenium.webdriver.remote.remote_connection] DEBUG: DELETE http://127.0.0.1:64533/session/5a6d8196d99d431b8b786f3f24688d84 {}
2021-11-03 13:03:00 [urllib3.connectionpool] DEBUG: http://127.0.0.1:64533 "DELETE /session/5a6d8196d99d431b8b786f3f24688d84 HTTP/1.1" 200 14
2021-11-03 13:03:00 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2021-11-03 13:03:02 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'elapsed_time_seconds': 0.005515,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2021, 11, 3, 19, 3, 0, 889623),
'log_count/DEBUG': 7,
'log_count/INFO': 10,
'start_time': datetime.datetime(2021, 11, 3, 19, 3, 0, 884108)}
2021-11-03 13:03:02 [scrapy.core.engine] INFO: Spider closed (finished)
(base) PS C:\Users\Cesal\projects\real_state\real_state\spiders>
我认为你的错误是你试图解析而不是启动请求。
变化：
def parse(self, response):
yield SeleniumRequest(
url='https://www.vivanuncios.com.mx/s-venta-inmuebles/queretaro/page-2/v1c1097l1021p2',
wait_time=3,
callback=self.parse
)
至：
def start_requests(self):
yield SeleniumRequest(
url='https://www.vivanuncios.com.mx/s-venta-inmuebles/queretaro/page-2/v1c1097l1021p2',
wait_time=3,
callback=self.parse
)
所以我尝试了几种方法来理解我的蜘蛛失败的原因，但没有成功。我已经被困了好几天了，不能再拖下去了。我只想抓取第一页，此时不进行分页。非常感谢您的帮助 :( 这是我的代码：
import scrapy
from scrapy_selenium import SeleniumRequest
class HomesSpider(scrapy.Spider):
name = 'homes'
def parse(self, response):
yield SeleniumRequest(
url='https://www.vivanuncios.com.mx/s-venta-inmuebles/queretaro/page-2/v1c1097l1021p2',
wait_time=3,
callback=self.parse
)
def parse(self, response):
homes = response.xpath("//div[@class='viewport-contents']/div")
for home in homes:
yield{
'price': home.xpath(".//span[@class='value wrapper']/span[@class='ad-price']/text()").get(),
'location': home.xpath(".//div[@class='tile-location one-liner']/b/text()").get(),
'description': home.xpath(".//div[@class='tile-desc one-liner']/a/text()").get(),
'bedrooms': home.xpath(".//div[@class='chiplets-inline-block re-bedroom']/text()").get(),
'm2': home.xpath(".//div[@class='chiplets-inline-block surface-area']/text()").get()
}
这是我的 settings.py 文件：
# Scrapy settings for real_state project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'real_state'
SPIDER_MODULES = ['real_state.spiders']
NEWSPIDER_MODULE = 'real_state.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'real_state (+http://www.yourdomain.com)'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
'User_Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36'
}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'real_state.middlewares.RealStateSpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
'scrapy_selenium.SeleniumMiddleware': 800
}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
# 'real_state.pipelines.RealStatePipeline': 300,
#}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
#SELENIUM
from shutil import which
SELENIUM_DRIVER_NAME = 'chrome'
SELENIUM_DRIVER_EXECUTABLE_PATH = which("C:\Users\Cesal\projects\real_state\chromedriver.exe")
SELENIUM_DRIVER_ARGUMENTS=['-headless'] # '--headless' if using chrome instead of firefox
这是我执行时在终端中得到的：
(base) PS C:\Users\Cesal\projects\real_state\real_state\spiders> scrapy crawl homes
2021-11-03 13:02:58 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: real_state)
2021-11-03 13:02:58 [scrapy.utils.log] INFO: Versions: lxml 4.6.1.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 21.2.0, Python 3.8.5 (default, Sep 3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1h 22 Sep 2020), cryptography 3.1.1, Platform Windows-10-10.0.19041-SP0
2021-11-03 13:02:58 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2021-11-03 13:02:58 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'real_state',
'NEWSPIDER_MODULE': 'real_state.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['real_state.spiders']}
2021-11-03 13:02:58 [scrapy.extensions.telnet] INFO: Telnet Password: ade49fc0492d5027
2021-11-03 13:02:58 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2021-11-03 13:02:59 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:64533/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "platformName": "any", "goog:chromeOptions": {"extensions": [], "args": ["-headless"]}}}, "desiredCapabilities": {"browserName": "chrome", "version": "", "platform": "ANY", "goog:chromeOptions": {"extensions": [], "args": ["-headless"]}}}
2021-11-03 13:02:59 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): 127.0.0.1:64533
DevTools listening on ws://127.0.0.1:64541/devtools/browser/302904be-ca13-4464-a332-8d995cb55f44
2021-11-03 13:03:00 [urllib3.connectionpool] DEBUG: http://127.0.0.1:64533 "POST /session HTTP/1.1" 200 788
2021-11-03 13:03:00 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2021-11-03 13:03:00 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy_selenium.SeleniumMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-11-03 13:03:00 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-11-03 13:03:00 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2021-11-03 13:03:00 [scrapy.core.engine] INFO: Spider opened
2021-11-03 13:03:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-11-03 13:03:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-11-03 13:03:00 [scrapy.core.engine] INFO: Closing spider (finished)
2021-11-03 13:03:00 [selenium.webdriver.remote.remote_connection] DEBUG: DELETE http://127.0.0.1:64533/session/5a6d8196d99d431b8b786f3f24688d84 {}
2021-11-03 13:03:00 [urllib3.connectionpool] DEBUG: http://127.0.0.1:64533 "DELETE /session/5a6d8196d99d431b8b786f3f24688d84 HTTP/1.1" 200 14
2021-11-03 13:03:00 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2021-11-03 13:03:02 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'elapsed_time_seconds': 0.005515,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2021, 11, 3, 19, 3, 0, 889623),
'log_count/DEBUG': 7,
'log_count/INFO': 10,
'start_time': datetime.datetime(2021, 11, 3, 19, 3, 0, 884108)}
2021-11-03 13:03:02 [scrapy.core.engine] INFO: Spider closed (finished)
(base) PS C:\Users\Cesal\projects\real_state\real_state\spiders>
我认为你的错误是你试图解析而不是启动请求。
变化：
def parse(self, response):
yield SeleniumRequest(
url='https://www.vivanuncios.com.mx/s-venta-inmuebles/queretaro/page-2/v1c1097l1021p2',
wait_time=3,
callback=self.parse
)
至：
def start_requests(self):
yield SeleniumRequest(
url='https://www.vivanuncios.com.mx/s-venta-inmuebles/queretaro/page-2/v1c1097l1021p2',
wait_time=3,
callback=self.parse
)