Scrapy 中间件设置

Question

我正在尝试使用 scrapy 访问 public 代理以获取一些数据。当我尝试运行代码时出现以下错误：

ImportError: Error loading object 'craiglist.middlewares.ProxyMiddleware': No module named middlewares

我使用以下代码创建了 middlewares.py 文件：

import base64
# Start your middleware class
class ProxyMiddleware(object):
    # overwrite process request
    def process_request(self, request, spider):
        # Set the location of the proxy
        request.meta['proxy'] = "http://124.200.36.150:8118"

settings.py:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
    'craiglist.middlewares.ProxyMiddleware': 100,
}

欢迎任何帮助。

Answer 1

使用此设置，您需要将 middlewares.py 上移到 craiglist 包中。

Scrapy 中间件设置

Scrapy middleware setup

python

web-crawler

scrapy

web-scraping