Type Error url is not JSON serializable in Scrapy with Selenium

Question

我在这里使用 python scrapy with selenium 我添加了代码

import scrapy
from examble.items import exambleItem
from selenium import webdriver


class exambleSpider(scrapy.Spider):
    name = "examble"
    allowed_domain = ['examble.com']
    start_urls = ['http://www.examble.com/kids/page-1.html']

    def parse(self,response):
        driver = webdriver.Chrome()
        driver.get(response)
        for i in range(1,36):
            item = ExambleItem()
            item['Name'] = 'sdds'       
            #driver = webdriver.Chrome()
            #driver.get(response)
            xpath = "//*[@id='wrapper']/main/div/div[2]/table/tbody/tr[%d]/td[1]/a" % i
            elem = driver.find_element_by_xpath(xpath)
            elem.click()
            yield item

我在运行代码中用scrapy crawl Type error发生错误是TypeError <200 http://www.examble.com/kids/page-1.html> is not JSON Serializable

我正在用 driver.get('http://www.examble.com/kids/page-1.html') 替换 driver.get(response) 它运行s 确切的错误应该是什么

Answer 1

response 是一个 scrapy.Response 对象。你在这里看到的是 response.url 因为 driver.get() 需要一个字符串参数。

只需将 response 替换为 response.url in:

driver.get(response)

Type Error url is not JSON serializable in Scrapy with Selenium

Type Error url is not JSON serializable in Scrapy with Selenium

selenium

scrapy

web-scraping

python-2.7

scrapy-spider