AttributeError: 'module' object has no attribute 'DATABASE' when using scrapy shell
AttributeError: 'module' object has no attribute 'DATABASE' when using scrapy shell
我正在尝试 运行 我项目根目录中的 scrapy shell,但我一直收到关于某种数据库设置的模糊错误。我不确定这是 SQLAlchemy 的问题……还是我的模式定义有问题?
如果我 运行 scrapy shell http://some_website.com
来自项目路径之外的任何其他目录,我没有问题。
正在尝试启动 shell:
me@me:~/my_spider$ scrapy shell http://some_website.com
2015-12-13 15:15:58-0800 [scrapy] INFO: Scrapy 0.24.4 started (bot: my_bot)
2015-12-13 15:15:58-0800 [scrapy] INFO: Optional features available: ssl, http11, boto, django
2015-12-13 15:15:58-0800 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'my_spider.spiders', 'DEPTH_LIMIT': 2, 'CONCURRENT_REQUESTS_PER_DOMAIN': 1, 'CONCURRENT_REQUESTS': 1, 'SPIDER_MODULES': [''my_spider.spiders'], 'BOT_NAME': 'my_bot', 'COOKIES_ENABLED': False, 'LOGSTATS_INTERVAL': 0, 'DOWNLOAD_DELAY': 5}
2015-12-13 15:15:58-0800 [scrapy] INFO: Enabled extensions: TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2015-12-13 15:15:59-0800 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, RandomUserAgentMiddleware, ProxyMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-12-13 15:15:59-0800 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
这是追溯:
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 11, in <module>
sys.exit(execute())
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 143, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 89, in _run_print_help
func(*a, **kw)
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 150, in _run_command
cmd.run(args, opts)
File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/shell.py", line 46, in run
self.crawler_process.start_crawling()
File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 124, in start_crawling
return self._start_crawler() is not None
File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 139, in _start_crawler
crawler.configure()
File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 47, in configure
self.engine = ExecutionEngine(self, self._spider_closed)
File "/usr/local/lib/python2.7/dist-packages/scrapy/core/engine.py", line 65, in __init__
self.scraper = Scraper(crawler)
File "/usr/local/lib/python2.7/dist-packages/scrapy/core/scraper.py", line 66, in __init__
self.itemproc = itemproc_cls.from_crawler(crawler)
File "/usr/local/lib/python2.7/dist-packages/scrapy/middleware.py", line 50, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "/usr/local/lib/python2.7/dist-packages/scrapy/middleware.py", line 35, in from_settings
mw = mwcls()
File "~/my_spider/pipelines.py", line 14, in __init__
engine = db_connect()
File "~/my_spider/libs/database/__init__.py", line 14, in db_connect
url = URL(**settings.DATABASE)
AttributeError: 'module' object has no attribute 'DATABASE'
如有任何建议,我们将不胜感激。
您有一个 settings
scrapy 找到的变量定义,而不是它认为会找到(或需要)的变量定义。
它不是查找 scrapy/middleware.py
模块 from_settings()
调用正在使用的 settings
对象,而是查找您的设置对象并希望它提供 .DATABASE
属性.没有看到 my_bot
中的代码,TraceBack 和最近的 Python 2.7 scrapy 中的代码行:
26 @classmethod
27 def from_settings(cls, settings, crawler=None):
28 mwlist = cls._get_mwlist_from_settings(settings)
29 middlewares = []
30 for clspath in mwlist:
31 try:
32 mwcls = load_object(clspath)
33 if crawler and hasattr(mwcls, 'from_crawler'):
34 mw = mwcls.from_crawler(crawler)
35 elif hasattr(mwcls, 'from_settings'):
36 mw = mwcls.from_settings(settings)
建议对您不想要的设置对象进行方法解析,或者您按照教程提供了 from_settings()
但未实现所需的属性。
除了@tristan 指出的内容之外,根据回溯 - 当您启动 shell 时,Scrapy 会获取您的项目设置,其中包括管道,其中一个正在执行 db_connect()
函数,它使用 settings.DATABASE
设置:
url = URL(**settings.DATABASE)
确保您在项目设置中定义了 DATABASE
字典。
我正在尝试 运行 我项目根目录中的 scrapy shell,但我一直收到关于某种数据库设置的模糊错误。我不确定这是 SQLAlchemy 的问题……还是我的模式定义有问题?
如果我 运行 scrapy shell http://some_website.com
来自项目路径之外的任何其他目录,我没有问题。
正在尝试启动 shell:
me@me:~/my_spider$ scrapy shell http://some_website.com
2015-12-13 15:15:58-0800 [scrapy] INFO: Scrapy 0.24.4 started (bot: my_bot)
2015-12-13 15:15:58-0800 [scrapy] INFO: Optional features available: ssl, http11, boto, django
2015-12-13 15:15:58-0800 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'my_spider.spiders', 'DEPTH_LIMIT': 2, 'CONCURRENT_REQUESTS_PER_DOMAIN': 1, 'CONCURRENT_REQUESTS': 1, 'SPIDER_MODULES': [''my_spider.spiders'], 'BOT_NAME': 'my_bot', 'COOKIES_ENABLED': False, 'LOGSTATS_INTERVAL': 0, 'DOWNLOAD_DELAY': 5}
2015-12-13 15:15:58-0800 [scrapy] INFO: Enabled extensions: TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2015-12-13 15:15:59-0800 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, RandomUserAgentMiddleware, ProxyMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-12-13 15:15:59-0800 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
这是追溯:
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 11, in <module>
sys.exit(execute())
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 143, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 89, in _run_print_help
func(*a, **kw)
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 150, in _run_command
cmd.run(args, opts)
File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/shell.py", line 46, in run
self.crawler_process.start_crawling()
File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 124, in start_crawling
return self._start_crawler() is not None
File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 139, in _start_crawler
crawler.configure()
File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 47, in configure
self.engine = ExecutionEngine(self, self._spider_closed)
File "/usr/local/lib/python2.7/dist-packages/scrapy/core/engine.py", line 65, in __init__
self.scraper = Scraper(crawler)
File "/usr/local/lib/python2.7/dist-packages/scrapy/core/scraper.py", line 66, in __init__
self.itemproc = itemproc_cls.from_crawler(crawler)
File "/usr/local/lib/python2.7/dist-packages/scrapy/middleware.py", line 50, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "/usr/local/lib/python2.7/dist-packages/scrapy/middleware.py", line 35, in from_settings
mw = mwcls()
File "~/my_spider/pipelines.py", line 14, in __init__
engine = db_connect()
File "~/my_spider/libs/database/__init__.py", line 14, in db_connect
url = URL(**settings.DATABASE)
AttributeError: 'module' object has no attribute 'DATABASE'
如有任何建议,我们将不胜感激。
您有一个 settings
scrapy 找到的变量定义,而不是它认为会找到(或需要)的变量定义。
它不是查找 scrapy/middleware.py
模块 from_settings()
调用正在使用的 settings
对象,而是查找您的设置对象并希望它提供 .DATABASE
属性.没有看到 my_bot
中的代码,TraceBack 和最近的 Python 2.7 scrapy 中的代码行:
26 @classmethod
27 def from_settings(cls, settings, crawler=None):
28 mwlist = cls._get_mwlist_from_settings(settings)
29 middlewares = []
30 for clspath in mwlist:
31 try:
32 mwcls = load_object(clspath)
33 if crawler and hasattr(mwcls, 'from_crawler'):
34 mw = mwcls.from_crawler(crawler)
35 elif hasattr(mwcls, 'from_settings'):
36 mw = mwcls.from_settings(settings)
建议对您不想要的设置对象进行方法解析,或者您按照教程提供了 from_settings()
但未实现所需的属性。
除了@tristan 指出的内容之外,根据回溯 - 当您启动 shell 时,Scrapy 会获取您的项目设置,其中包括管道,其中一个正在执行 db_connect()
函数,它使用 settings.DATABASE
设置:
url = URL(**settings.DATABASE)
确保您在项目设置中定义了 DATABASE
字典。