Scrapy 爬取重复数据
Scrapy crawls duplicate data
不幸的是，我目前在使用 Scrapy 时遇到了问题。我对Scrapy还是个新手，想报废劳力士手表的信息。我从网站 Watch.de 开始，我首先浏览劳力士网站并想打开个别手表以获取确切信息。但是，当我启动爬虫时，我看到很多手表都被爬了好几次。我假设这些是来自“最近查看”和“我们的新品”点的手表。有没有办法忽略这些重复项？
这是我的代码
class WatchbotSpider(scrapy.Spider):
name = 'watchbot'
start_urls = ['https://www.watch.de/germany/rolex.html']
def parse(self, response, **kwargs):
for link in response.css('div.product-item-link a::attr(href)'):
yield response.follow(link.get(), callback=self.parse_categories)
def parse_categories(self, response):
for product in response.css('div.product-item-link'):
yield {
'id': product.css('span.product-item-id.product-item-ref::text').get(),
'brand': product.css('div.product-item-brand::text').get(),
'model': product.css('div.product-item-model::text').get(),
'price': product.css('span.price::text').get(),
'year': product.css('span.product-item-id.product-item-year::text').get()
这有效，
import scrapy
class WatchbotSpider(scrapy.Spider):
name = 'watchbot'
start_urls = ['https://www.watch.de/germany/rolex.html']
def parse(self, response, **kwargs):
for link in response.css('div.product-item-link a::attr(href)'):
yield response.follow(link.get(), callback=self.parse_categories)
def parse_categories(self, response):
Dict = {
'id': response.xpath('//div[@class="product-ref-item product-ref d-flex align-items-center"]/span/text()').get(),
'brand': response.css('div.product-item-brand::text').get(),
'model': response.xpath('//h1[@class="product-name"]/text()').get(),
'price': response.css('span.price::text').get().replace(u'\xa0', u' '),
'year': response.xpath('//div[@class="product-item-date product-item-option"]/span/text()').get(),
}
print(Dict)
yield Dict
scrapy crawl watchbot > log
在日志中，
{'id': '278240', 'brand': 'Rolex ', 'model': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Oyster - 31mm - Ungetragen ', 'price': '8.118 €', 'year': '2021'}
{'id': '116201', 'brand': 'Rolex', 'model': 'Rolex Datejust - Stahl / Roségold - Armband Stahl / Roségold / Oyster - 36mm - Wie neu ', 'price': '14.545 €', 'year': '2018'}
{'id': '126622', 'brand': 'Rolex', 'model': 'Rolex Yacht-Master - Stahl / Platin - Armband Edelstahl / Oyster - 40mm - Ungetragen ', 'price': '15.995 €', 'year': '2020'}
{'id': '124300', 'brand': 'Rolex', 'model': 'Rolex Oyster Perpetual - Edelstahl - Armband Edelstahl / Oyster - 41mm - Ungetragen ', 'price': '9.898 €', 'year': '2021'}
{'id': '116500LN', 'brand': 'Rolex', 'model': 'Rolex Daytona - Edelstahl - Armband Edelstahl / Oyster - 40mm - Wie neu ', 'price': '33.999 €', 'year': '2020'}
{'id': '115234', 'brand': 'Rolex', 'model': 'Rolex Oyster Perpetual Date Diamanten - Stahl / Weißgold - Armband Edelstahl / Oyster - 34mm - Ungetragen - Vintage ', 'price': '11.990 €', 'year': '2021'}
{'id': '126200', 'brand': 'Rolex', 'model': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Jubilé - 36mm - Ungetragen ', 'price': '9.595 €', 'year': '2021'}
{'id': '126333 ', 'brand': 'Rolex', 'model': 'Rolex Datejust - Stahl / Gelbgold - Armband Stahl / Gelbgold / Jubilé - 41mm - Wie neu ', 'price': '15.959 €', 'year': '2021'}
{'id': '126334 ', 'brand': 'Rolex', 'model': 'Rolex Datejust Wimbledon - Stahl / Weißgold - Armband Edelstahl / Oyster - 41mm - Ungetragen ', 'price': '13.399 €', 'year': '2021'}
{'id': '278240', 'brand': 'Rolex', 'model': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Oyster - 31mm - Ungetragen ', 'price': '8.118 €', 'year': '2021'}
.
.
.
格式化 replace(" ", "") 会导致一些异常，因此下一步要小心格式化。
因为您正在迭代每个项目 link 这就是为什么不需要再次重申 lop 的含义并且您必须将个人页面与个人 url 一起带到个人页面并且从该页面您必须 select你想要的数据项。
代码：
import scrapy
class WatchbotSpider(scrapy.Spider):
name = 'watchbot'
start_urls = ['https://www.watch.de/germany/rolex.html']
def parse(self, response, **kwargs):
for link in response.css('div.product-item-link a::attr(href)'):
url = link.get()
yield scrapy.Request(url, callback=self.parse_categories)
def parse_categories(self, response):
yield {
'id': response.css('div.product-ref-item.product-sku.col-auto>span[itemprop="sku"]::text').get(),
'product_name': response.css('h1.product-name::text').get().strip(),
'price': response.xpath('(.//span[@class="price"])[1]/text()').get().replace('\xa0€',' '),
'year': response.xpath('.//*[@class="product-item-date product-item-option"]/span/text()').get()}
输出：
{'id': '10000060888', 'product_name': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Oyster
- 31mm - Ungetragen', 'price': '8.118 ', 'year': '2021'}
2021-10-26 18:42:53 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-oyster-perpetual-date-stahl-weissgold-diamanten-automatik-armband-oyster-34mm-ref-115234-vintage-bj-2021-box-pap-full-set-ungetragen-verklebt.html>
{'id': '10000060571', 'product_name': 'Rolex Oyster Perpetual Date Diamanten - Stahl / Weißgold
- Armband Edelstahl / Oyster - 34mm - Ungetragen - Vintage', 'price': '11.990 ', 'year': '2021'}
2021-10-26 18:42:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-sky-dweller-stahl-gelbgold-automatik-armband-stahl-gelbgold-jubile-42mm-ref-326933-bj-2021-box-pap-full-set-ungetragen-neuheit-2021.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:53 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-sky-dweller-stahl-gelbgold-automatik-armband-stahl-gelbgold-jubile-42mm-ref-326933-bj-2021-box-pap-full-set-ungetragen-neuheit-2021.html>
{'id': '10000060597', 'product_name': 'Rolex Sky-Dweller - Stahl / Gelbgold - Armband Stahl / Gelbgold / Jubilé - 42mm - Ungetragen', 'price': '24.924 ', 'year': '2021'}
2021-10-26 18:42:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-yacht-master-ii-stahl-rosegold-everose-automatik-chronograph-44mm-ref-116681-bj-2021-box-pap-full-set-ungetragen.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-datejust-stahl-automatik-armband-oyster-36mm-ref-126200-box-pap-lc-eu-full-set-wie-neu.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:54 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-yacht-master-ii-stahl-rosegold-everose-automatik-chronograph-44mm-ref-116681-bj-2021-box-pap-full-set-ungetragen.html>
{'id': '10000060591', 'product_name': 'Rolex Yacht-Master II - Stahl / Roségold - Armband Stahl
/ Roségold / Oyster - 44mm - Ungetragen', 'price': '27.979 ', 'year': '2021'}
2021-10-26 18:42:54 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-datejust-stahl-automatik-armband-oyster-36mm-ref-126200-box-pap-lc-eu-full-set-wie-neu.html>
{'id': '10000060580', 'product_name': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Oyster - 36mm - Wie neu', 'price': '8.080 ', 'year': '2015'}
2021-10-26 18:42:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-datejust-medium-stahl-automatik-armband-oyster-31mm-ref-278240-bj-2021-box-pap-full-set-ungetragen-60579.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-datejust-stahl-automatik-armband-oyster-41mm-ref-126300-box-pap-full-set-ungetragen-60647.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-datejust-medium-stahl-automatik-armband-oyster-31mm-ref-278240-bj-2021-box-pap-full-set-ungetragen-60579.html>
{'id': '10000060579', 'product_name': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Oyster
- 31mm - Ungetragen', 'price': '8.118 ', 'year': '2021'}
2021-10-26 18:42:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-submariner-stahl-keramik-automatik-armband-oyster-40mm-ref-114060-bj-2015-box-pap-lc100-full-set-wie-neu.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-datejust-stahl-automatik-armband-oyster-41mm-ref-126300-box-pap-full-set-ungetragen-60647.html>
{'id': '10000060647', 'product_name': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Oyster - 41mm - Ungetragen', 'price': '9.999 ', 'year': '2021'}
2021-10-26 18:42:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-submariner-stahl-keramik-automatik-armband-oyster-40mm-ref-114060-bj-2015-box-pap-lc100-full-set-wie-neu.html>
{'id': '10000060625', 'product_name': 'Rolex Submariner - Edelstahl - Armband Edelstahl / Oyster - 40mm - Wie neu', 'price': '12.525 ', 'year': '2015'}
2021-10-26 18:42:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-oyster-perpetual-tiffany-stahl-automatik-armband-oyster-41mm-ref-124300-box-pap-full-set-ungetragen-60598.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-oyster-perpetual-tiffany-stahl-automatik-armband-oyster-41mm-ref-124300-box-pap-full-set-ungetragen-60598.html>
{'id': '10000060598', 'product_name': 'Rolex Oyster Perpetual Tiffany - Edelstahl - Armband Edelstahl / Oyster - 41mm - Ungetragen', 'price': '16.565 ', 'year': '2021'}
2021-10-26 18:42:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-datejust-41-stahl-weissgold-automatik-armband-jubile-41mm-ref-126334-box-pap-full-set-ungetragen-60614.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-sea-dweller-red-4000-stahl-keramik-automatik-armband-oyster-43mm-ref-126600-bj-2018-box-pap-lc-eu-full-set-wie-neu.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:56 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-datejust-41-stahl-weissgold-automatik-armband-jubile-41mm-ref-126334-box-pap-full-set-ungetragen-60614.html>
等等
不幸的是，我目前在使用 Scrapy 时遇到了问题。我对Scrapy还是个新手，想报废劳力士手表的信息。我从网站 Watch.de 开始，我首先浏览劳力士网站并想打开个别手表以获取确切信息。但是，当我启动爬虫时，我看到很多手表都被爬了好几次。我假设这些是来自“最近查看”和“我们的新品”点的手表。有没有办法忽略这些重复项？
这是我的代码
class WatchbotSpider(scrapy.Spider):
name = 'watchbot'
start_urls = ['https://www.watch.de/germany/rolex.html']
def parse(self, response, **kwargs):
for link in response.css('div.product-item-link a::attr(href)'):
yield response.follow(link.get(), callback=self.parse_categories)
def parse_categories(self, response):
for product in response.css('div.product-item-link'):
yield {
'id': product.css('span.product-item-id.product-item-ref::text').get(),
'brand': product.css('div.product-item-brand::text').get(),
'model': product.css('div.product-item-model::text').get(),
'price': product.css('span.price::text').get(),
'year': product.css('span.product-item-id.product-item-year::text').get()
这有效，
import scrapy
class WatchbotSpider(scrapy.Spider):
name = 'watchbot'
start_urls = ['https://www.watch.de/germany/rolex.html']
def parse(self, response, **kwargs):
for link in response.css('div.product-item-link a::attr(href)'):
yield response.follow(link.get(), callback=self.parse_categories)
def parse_categories(self, response):
Dict = {
'id': response.xpath('//div[@class="product-ref-item product-ref d-flex align-items-center"]/span/text()').get(),
'brand': response.css('div.product-item-brand::text').get(),
'model': response.xpath('//h1[@class="product-name"]/text()').get(),
'price': response.css('span.price::text').get().replace(u'\xa0', u' '),
'year': response.xpath('//div[@class="product-item-date product-item-option"]/span/text()').get(),
}
print(Dict)
yield Dict
scrapy crawl watchbot > log
在日志中，
{'id': '278240', 'brand': 'Rolex ', 'model': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Oyster - 31mm - Ungetragen ', 'price': '8.118 €', 'year': '2021'}
{'id': '116201', 'brand': 'Rolex', 'model': 'Rolex Datejust - Stahl / Roségold - Armband Stahl / Roségold / Oyster - 36mm - Wie neu ', 'price': '14.545 €', 'year': '2018'}
{'id': '126622', 'brand': 'Rolex', 'model': 'Rolex Yacht-Master - Stahl / Platin - Armband Edelstahl / Oyster - 40mm - Ungetragen ', 'price': '15.995 €', 'year': '2020'}
{'id': '124300', 'brand': 'Rolex', 'model': 'Rolex Oyster Perpetual - Edelstahl - Armband Edelstahl / Oyster - 41mm - Ungetragen ', 'price': '9.898 €', 'year': '2021'}
{'id': '116500LN', 'brand': 'Rolex', 'model': 'Rolex Daytona - Edelstahl - Armband Edelstahl / Oyster - 40mm - Wie neu ', 'price': '33.999 €', 'year': '2020'}
{'id': '115234', 'brand': 'Rolex', 'model': 'Rolex Oyster Perpetual Date Diamanten - Stahl / Weißgold - Armband Edelstahl / Oyster - 34mm - Ungetragen - Vintage ', 'price': '11.990 €', 'year': '2021'}
{'id': '126200', 'brand': 'Rolex', 'model': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Jubilé - 36mm - Ungetragen ', 'price': '9.595 €', 'year': '2021'}
{'id': '126333 ', 'brand': 'Rolex', 'model': 'Rolex Datejust - Stahl / Gelbgold - Armband Stahl / Gelbgold / Jubilé - 41mm - Wie neu ', 'price': '15.959 €', 'year': '2021'}
{'id': '126334 ', 'brand': 'Rolex', 'model': 'Rolex Datejust Wimbledon - Stahl / Weißgold - Armband Edelstahl / Oyster - 41mm - Ungetragen ', 'price': '13.399 €', 'year': '2021'}
{'id': '278240', 'brand': 'Rolex', 'model': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Oyster - 31mm - Ungetragen ', 'price': '8.118 €', 'year': '2021'}
.
.
.
格式化 replace(" ", "") 会导致一些异常，因此下一步要小心格式化。
因为您正在迭代每个项目 link 这就是为什么不需要再次重申 lop 的含义并且您必须将个人页面与个人 url 一起带到个人页面并且从该页面您必须 select你想要的数据项。
代码：
import scrapy
class WatchbotSpider(scrapy.Spider):
name = 'watchbot'
start_urls = ['https://www.watch.de/germany/rolex.html']
def parse(self, response, **kwargs):
for link in response.css('div.product-item-link a::attr(href)'):
url = link.get()
yield scrapy.Request(url, callback=self.parse_categories)
def parse_categories(self, response):
yield {
'id': response.css('div.product-ref-item.product-sku.col-auto>span[itemprop="sku"]::text').get(),
'product_name': response.css('h1.product-name::text').get().strip(),
'price': response.xpath('(.//span[@class="price"])[1]/text()').get().replace('\xa0€',' '),
'year': response.xpath('.//*[@class="product-item-date product-item-option"]/span/text()').get()}
输出：
{'id': '10000060888', 'product_name': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Oyster
- 31mm - Ungetragen', 'price': '8.118 ', 'year': '2021'}
2021-10-26 18:42:53 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-oyster-perpetual-date-stahl-weissgold-diamanten-automatik-armband-oyster-34mm-ref-115234-vintage-bj-2021-box-pap-full-set-ungetragen-verklebt.html>
{'id': '10000060571', 'product_name': 'Rolex Oyster Perpetual Date Diamanten - Stahl / Weißgold
- Armband Edelstahl / Oyster - 34mm - Ungetragen - Vintage', 'price': '11.990 ', 'year': '2021'}
2021-10-26 18:42:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-sky-dweller-stahl-gelbgold-automatik-armband-stahl-gelbgold-jubile-42mm-ref-326933-bj-2021-box-pap-full-set-ungetragen-neuheit-2021.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:53 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-sky-dweller-stahl-gelbgold-automatik-armband-stahl-gelbgold-jubile-42mm-ref-326933-bj-2021-box-pap-full-set-ungetragen-neuheit-2021.html>
{'id': '10000060597', 'product_name': 'Rolex Sky-Dweller - Stahl / Gelbgold - Armband Stahl / Gelbgold / Jubilé - 42mm - Ungetragen', 'price': '24.924 ', 'year': '2021'}
2021-10-26 18:42:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-yacht-master-ii-stahl-rosegold-everose-automatik-chronograph-44mm-ref-116681-bj-2021-box-pap-full-set-ungetragen.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-datejust-stahl-automatik-armband-oyster-36mm-ref-126200-box-pap-lc-eu-full-set-wie-neu.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:54 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-yacht-master-ii-stahl-rosegold-everose-automatik-chronograph-44mm-ref-116681-bj-2021-box-pap-full-set-ungetragen.html>
{'id': '10000060591', 'product_name': 'Rolex Yacht-Master II - Stahl / Roségold - Armband Stahl
/ Roségold / Oyster - 44mm - Ungetragen', 'price': '27.979 ', 'year': '2021'}
2021-10-26 18:42:54 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-datejust-stahl-automatik-armband-oyster-36mm-ref-126200-box-pap-lc-eu-full-set-wie-neu.html>
{'id': '10000060580', 'product_name': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Oyster - 36mm - Wie neu', 'price': '8.080 ', 'year': '2015'}
2021-10-26 18:42:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-datejust-medium-stahl-automatik-armband-oyster-31mm-ref-278240-bj-2021-box-pap-full-set-ungetragen-60579.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-datejust-stahl-automatik-armband-oyster-41mm-ref-126300-box-pap-full-set-ungetragen-60647.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-datejust-medium-stahl-automatik-armband-oyster-31mm-ref-278240-bj-2021-box-pap-full-set-ungetragen-60579.html>
{'id': '10000060579', 'product_name': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Oyster
- 31mm - Ungetragen', 'price': '8.118 ', 'year': '2021'}
2021-10-26 18:42:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-submariner-stahl-keramik-automatik-armband-oyster-40mm-ref-114060-bj-2015-box-pap-lc100-full-set-wie-neu.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-datejust-stahl-automatik-armband-oyster-41mm-ref-126300-box-pap-full-set-ungetragen-60647.html>
{'id': '10000060647', 'product_name': 'Rolex Datejust - Edelstahl - Armband Edelstahl / Oyster - 41mm - Ungetragen', 'price': '9.999 ', 'year': '2021'}
2021-10-26 18:42:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-submariner-stahl-keramik-automatik-armband-oyster-40mm-ref-114060-bj-2015-box-pap-lc100-full-set-wie-neu.html>
{'id': '10000060625', 'product_name': 'Rolex Submariner - Edelstahl - Armband Edelstahl / Oyster - 40mm - Wie neu', 'price': '12.525 ', 'year': '2015'}
2021-10-26 18:42:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-oyster-perpetual-tiffany-stahl-automatik-armband-oyster-41mm-ref-124300-box-pap-full-set-ungetragen-60598.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-oyster-perpetual-tiffany-stahl-automatik-armband-oyster-41mm-ref-124300-box-pap-full-set-ungetragen-60598.html>
{'id': '10000060598', 'product_name': 'Rolex Oyster Perpetual Tiffany - Edelstahl - Armband Edelstahl / Oyster - 41mm - Ungetragen', 'price': '16.565 ', 'year': '2021'}
2021-10-26 18:42:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-datejust-41-stahl-weissgold-automatik-armband-jubile-41mm-ref-126334-box-pap-full-set-ungetragen-60614.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.watch.de/germany/rolex-sea-dweller-red-4000-stahl-keramik-automatik-armband-oyster-43mm-ref-126600-bj-2018-box-pap-lc-eu-full-set-wie-neu.html> (referer: https://www.watch.de/germany/rolex.html)
2021-10-26 18:42:56 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.watch.de/germany/rolex-datejust-41-stahl-weissgold-automatik-armband-jubile-41mm-ref-126334-box-pap-full-set-ungetragen-60614.html>
等等