Django Celery 任务 运行 两次
Django Celery tasks run twice
我的项目目录结构如下：
rss_reader
- reader
- __init__.py
- models.py
- views.py
- ...
- rss_reader
- __init__.py
- settings.py
- urls.py
- wsgi.py
- rss_contents
- __init__.py
- celery.py
- tasks.py
- config.py
- ...
tasks.py
from __future__ import absolute_import
from rss_contents.celery import app
@app.task
def processArticle():
print "100"
celery.py
from __future__ import absolute_import
from celery import Celery
app = Celery('rss_contents', include=['rss_contents.tasks'])
app.config_from_object('rss_contents.config')
config.py
from __future__ import absolute_import
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/5'
BROKER_URL = 'redis://127.0.0.1:6379/6'
reader > __init__.py
from rss_contents import tasks
tasks.processArticle.delay()
我的步骤如下：
- celery -A rss_contents worker -l info
控制台显示：
[2017-01-11 10:34:33,829: INFO/MainProcess] Connected to redis://127.0.0.1:6379/6
[2017-01-11 10:34:33,855: INFO/MainProcess] mingle: searching for neighbors
[2017-01-11 10:34:34,861: INFO/MainProcess] mingle: all alone
[2017-01-11 10:34:34,892: WARNING/MainProcess] celery@DESKTOP-6KAT7MF ready.
- 运行 服务器，在 http://127.0.0.1:8000/
启动开发服务器
但是celery控制台显示任务是运行两次：
[2017-01-11 10:41:20,910: INFO/MainProcess] Received task: rss_contents.tasks.processArticle[aa355c77-4ee8-4208-9e8c-915b110c7bbd]
[2017-01-11 10:41:20,911: WARNING/Worker-1] 100
[2017-01-11 10:41:20,917: INFO/MainProcess] Task rss_contents.tasks.processArticle[aa355c77-4ee8-4208-9e8c-915b110c7bbd] succeeded in 0.00600004196167s: None
[2017-01-11 10:41:22,430: INFO/MainProcess] Received task: rss_contents.tasks.processArticle[d92a151c-f0f9-4e8f-921a-fff2c1eb64c6]
[2017-01-11 10:41:22,431: WARNING/Worker-1] 100
[2017-01-11 10:41:22,447: INFO/MainProcess] Task rss_contents.tasks.processArticle[d92a151c-f0f9-4e8f-921a-fff2c1eb64c6] succeeded in 0.0160000324249s: None
为什么运行宁了两次？一次怎么办？
提前致谢。
那是因为你在__init__.py文件中定义了任务调用。
我已经开始用你的代码进行测试，并像这样在 __init__.py 中插入一个 pdb 中断
from celery_proj import tasks
import pdb; pdb.set_trace() # <-- make a break point here
print __file__
tasks.processArticle.delay()
并获得以下兴趣发现：
当您 运行 python manage.py runserver 时，断点触发一次，堆栈跟踪如下：
(Pdb) where
/Users/enix/Source/python/Whosebug/sotest/manage.py(10)<module>()
-> execute_from_command_line(sys.argv)
/Users/enix/Source/python/bae/lib/python2.7/site-packages/django/core/management/__init__.py(353)execute_from_command_line()
-> utility.execute()
/Users/enix/Source/python/bae/lib/python2.7/site-packages/django/core/management/__init__.py(316)execute()
-> autoreload.check_errors(django.setup)()
/Users/enix/Source/python/bae/lib/python2.7/site-packages/django/utils/autoreload.py(226)wrapper()
-> fn(*args, **kwargs)
/Users/enix/Source/python/bae/lib/python2.7/site-packages/django/__init__.py(18)setup()
-> apps.populate(settings.INSTALLED_APPS)
/Users/enix/Source/python/bae/lib/python2.7/site-packages/django/apps/registry.py(85)populate()
-> app_config = AppConfig.create(entry)
/Users/enix/Source/python/bae/lib/python2.7/site-packages/django/apps/config.py(90)create()
-> module = import_module(entry)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py(37)import_module()
-> __import__(name)
> /Users/enix/Source/python/Whosebug/sotest/app/__init__.py(4)<module>()
-> print __file__
并且当我键入 c 继续时，断点再次触发并使用相同的堆栈跟踪。
所以你的任务被调用了两次。但是不太清楚为什么django会导入两次应用包。
(bae)bogon:sotest enix$ python manage.py runserver
/Users/enix/Source/python/Whosebug/sotest/app/__init__.py # <-- print __file__
/Users/enix/Source/python/Whosebug/sotest/app/__init__.pyc # <-- print __file__
Performing system checks...
System check identified no issues (0 silenced).
January 10, 2017 - 21:44:12
Django version 1.9.12, using settings 'sotest.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
也许你应该把你的任务调用移到其他地方再试一次。
@enix 把它钉在了头上。因为reader是一个app，所以在django初始化应用注册表的时候会调用一次。但是，由于此调用是动态进行的，因此 reader 应用程序可能会被第二次初始化，因为在某处全局使用了 reader 包。在任何情况下，解决此问题的最简单方法就是将调用移至 class 并从任何 python 文件 中的单例 调用它 .
class AppInitializer(object):
initialized = false
@classmethod
def initialize(cls):
if not cls.initialized:
cls.initialized = True
from rss_contents import tasks
tasks.processArticle.delay()
AppInitializer.initialize()
如果您的应用引擎是多线程的，您需要在初始化周围添加一个方法锁。
我的项目目录结构如下：
rss_reader
- reader
- __init__.py
- models.py
- views.py
- ...
- rss_reader
- __init__.py
- settings.py
- urls.py
- wsgi.py
- rss_contents
- __init__.py
- celery.py
- tasks.py
- config.py
- ...
tasks.py
from __future__ import absolute_import
from rss_contents.celery import app
@app.task
def processArticle():
print "100"
celery.py
from __future__ import absolute_import
from celery import Celery
app = Celery('rss_contents', include=['rss_contents.tasks'])
app.config_from_object('rss_contents.config')
config.py
from __future__ import absolute_import
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/5'
BROKER_URL = 'redis://127.0.0.1:6379/6'
reader > __init__.py
from rss_contents import tasks
tasks.processArticle.delay()
我的步骤如下：
- celery -A rss_contents worker -l info
控制台显示：
[2017-01-11 10:34:33,829: INFO/MainProcess] Connected to redis://127.0.0.1:6379/6
[2017-01-11 10:34:33,855: INFO/MainProcess] mingle: searching for neighbors
[2017-01-11 10:34:34,861: INFO/MainProcess] mingle: all alone
[2017-01-11 10:34:34,892: WARNING/MainProcess] celery@DESKTOP-6KAT7MF ready.
- 运行 服务器，在 http://127.0.0.1:8000/ 启动开发服务器
但是celery控制台显示任务是运行两次：
[2017-01-11 10:41:20,910: INFO/MainProcess] Received task: rss_contents.tasks.processArticle[aa355c77-4ee8-4208-9e8c-915b110c7bbd]
[2017-01-11 10:41:20,911: WARNING/Worker-1] 100
[2017-01-11 10:41:20,917: INFO/MainProcess] Task rss_contents.tasks.processArticle[aa355c77-4ee8-4208-9e8c-915b110c7bbd] succeeded in 0.00600004196167s: None
[2017-01-11 10:41:22,430: INFO/MainProcess] Received task: rss_contents.tasks.processArticle[d92a151c-f0f9-4e8f-921a-fff2c1eb64c6]
[2017-01-11 10:41:22,431: WARNING/Worker-1] 100
[2017-01-11 10:41:22,447: INFO/MainProcess] Task rss_contents.tasks.processArticle[d92a151c-f0f9-4e8f-921a-fff2c1eb64c6] succeeded in 0.0160000324249s: None
为什么运行宁了两次？一次怎么办？
提前致谢。
那是因为你在__init__.py文件中定义了任务调用。
我已经开始用你的代码进行测试，并像这样在 __init__.py 中插入一个 pdb 中断
from celery_proj import tasks
import pdb; pdb.set_trace() # <-- make a break point here
print __file__
tasks.processArticle.delay()
并获得以下兴趣发现：
当您 运行 python manage.py runserver 时，断点触发一次，堆栈跟踪如下：
(Pdb) where
/Users/enix/Source/python/Whosebug/sotest/manage.py(10)<module>()
-> execute_from_command_line(sys.argv)
/Users/enix/Source/python/bae/lib/python2.7/site-packages/django/core/management/__init__.py(353)execute_from_command_line()
-> utility.execute()
/Users/enix/Source/python/bae/lib/python2.7/site-packages/django/core/management/__init__.py(316)execute()
-> autoreload.check_errors(django.setup)()
/Users/enix/Source/python/bae/lib/python2.7/site-packages/django/utils/autoreload.py(226)wrapper()
-> fn(*args, **kwargs)
/Users/enix/Source/python/bae/lib/python2.7/site-packages/django/__init__.py(18)setup()
-> apps.populate(settings.INSTALLED_APPS)
/Users/enix/Source/python/bae/lib/python2.7/site-packages/django/apps/registry.py(85)populate()
-> app_config = AppConfig.create(entry)
/Users/enix/Source/python/bae/lib/python2.7/site-packages/django/apps/config.py(90)create()
-> module = import_module(entry)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py(37)import_module()
-> __import__(name)
> /Users/enix/Source/python/Whosebug/sotest/app/__init__.py(4)<module>()
-> print __file__
并且当我键入 c 继续时，断点再次触发并使用相同的堆栈跟踪。
所以你的任务被调用了两次。但是不太清楚为什么django会导入两次应用包。
(bae)bogon:sotest enix$ python manage.py runserver
/Users/enix/Source/python/Whosebug/sotest/app/__init__.py # <-- print __file__
/Users/enix/Source/python/Whosebug/sotest/app/__init__.pyc # <-- print __file__
Performing system checks...
System check identified no issues (0 silenced).
January 10, 2017 - 21:44:12
Django version 1.9.12, using settings 'sotest.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
也许你应该把你的任务调用移到其他地方再试一次。
@enix 把它钉在了头上。因为reader是一个app，所以在django初始化应用注册表的时候会调用一次。但是，由于此调用是动态进行的，因此 reader 应用程序可能会被第二次初始化，因为在某处全局使用了 reader 包。在任何情况下，解决此问题的最简单方法就是将调用移至 class 并从任何 python 文件 中的单例 调用它 .
class AppInitializer(object):
initialized = false
@classmethod
def initialize(cls):
if not cls.initialized:
cls.initialized = True
from rss_contents import tasks
tasks.processArticle.delay()
AppInitializer.initialize()
如果您的应用引擎是多线程的，您需要在初始化周围添加一个方法锁。