Celery Beat 调度程序不会在 Django 中执行我的任务
Celery Beat scheduler won't execute my task in Django
我在 Django 中设置 celery 模块时遇到了一些问题。我不仅对设置要求很困惑，而且我也很难理解我的错误。
我想在每天的特定时间运行执行一个函数，这是我以前用 crontab 做的。 Celery 显然更适合这种类型的任务，这就是为什么我试图转向这个解决方案。
设置似乎没问题，当我调用它时我看到了 celery 调度程序 运行，但是当间隔开始时我的任务没有被处理。
我看到了很多（!!!）芹菜的不同设置，所以我的代码可能包含冗余和不必要的行。
我要触发的函数称为 team_accountabillity() 并且存在于我的应用程序中 "myteam.task.py"
我有 Celery 4.4.0，我使用 rabbitMQ 作为队列管理器（我还安装了 redis，因为很多教程都用这个）。
这是我的 proj/settings.py
from __future__ import absolute_import
import os
from datetime import timedelta
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_BEAT_SCHEDULER = {
'team accountability': {
'task':'team_accountability', 'schedule': timedelta(seconds=10),
},
}
CELERY_TIMEZONE = 'UTC'
CELERY_ENABLE_UTC = True
INSTALLED_APPS = [
'celerybeat_status',
'django_celery_beat',
'myteam.apps.MyteamConfig',
]
proj/celery.py 文件：
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from celery.schedules import crontab
from django.apps import apps
from django.conf import settings
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
app = Celery('proj')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks(lambda: [n.name for n in apps.get_app_configs()])
##documentation regarding the next line:
##https://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#beat-entries
##I find it very weird that this command is very similar to the CELERY_BEAT_SCHEDULER in settings.py
##I honestly don't know what this does
@app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
sender.add_periodic_task(10.0, team_accountability, name='team accountability')
@app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
proj/init.py文件:
from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app
__all__ = ('celery_app')
最后，这是我的 app/task.py (myteam/task.py)
from __future__ import absolute_import, unicode_literals
from celery import shared_task
from myteam.models import Team
##decorator should be @shared_task or @task???
@shared_task
def team_accountability():
t_all = Team.objects.all()
for t in t_all:
total = 0
s_all = t.stocks.all()
for s in s_all:
symbol = s.symbol
queryset = Stock.objects.filter(symbol__contains=symbol).order_by("-id").first()
total = total + queryset.change_percent
t.total = round(total, 3)
t.save()
综上所述，这是我 运行 命令时得到的结果：
celery -A proj worker -l debug
我得到的是这样的：
[2020-02-21 12:39:03,420: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2020-02-21 12:39:03,420: DEBUG/MainProcess] | Worker: Building graph...
[2020-02-21 12:39:03,421: DEBUG/MainProcess] | Worker: New boot order: {StateDB, Beat, Timer, Hub, Pool, Autoscaler, Consumer}
[2020-02-21 12:39:03,429: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2020-02-21 12:39:03,429: DEBUG/MainProcess] | Consumer: Building graph...
[2020-02-21 12:39:03,447: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Gossip, Tasks, Control, Agent, Heart, event loop}
-------------- celery@matador-B450M-DS3H v4.4.0 (cliffs)
--- ***** -----
-- ******* ---- Linux-4.15.0-70-generic-x86_64-with-debian-buster-sid 2020-02-21 12:39:03
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: TheRodeoProject:0x7f00682ecc10
- ** ---------- .> transport: redis://localhost:6379//
- ** ---------- .> results: redis://localhost:6379/
- *** --- * --- .> concurrency: 12 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. TheRodeoProject.celery.debug_task
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
[2020-02-21 12:39:03,455: DEBUG/MainProcess] | Worker: Starting Hub
[2020-02-21 12:39:03,455: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:03,455: DEBUG/MainProcess] | Worker: Starting Pool
[2020-02-21 12:39:03,891: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:03,891: DEBUG/MainProcess] | Worker: Starting Consumer
[2020-02-21 12:39:03,891: DEBUG/MainProcess] | Consumer: Starting Connection
[2020-02-21 12:39:03,900: INFO/MainProcess] Connected to redis://localhost:6379//
[2020-02-21 12:39:03,901: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:03,901: DEBUG/MainProcess] | Consumer: Starting Events
[2020-02-21 12:39:03,908: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:03,908: DEBUG/MainProcess] | Consumer: Starting Mingle
[2020-02-21 12:39:03,908: INFO/MainProcess] mingle: searching for neighbors
[2020-02-21 12:39:04,924: INFO/MainProcess] mingle: all alone
[2020-02-21 12:39:04,924: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:04,925: DEBUG/MainProcess] | Consumer: Starting Gossip
[2020-02-21 12:39:04,930: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:04,930: DEBUG/MainProcess] | Consumer: Starting Tasks
[2020-02-21 12:39:04,934: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:04,934: DEBUG/MainProcess] | Consumer: Starting Control
[2020-02-21 12:39:04,935: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:04,935: DEBUG/MainProcess] | Consumer: Starting Heart
[2020-02-21 12:39:04,936: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:04,936: DEBUG/MainProcess] | Consumer: Starting event loop
[2020-02-21 12:39:04,937: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2020-02-21 12:39:04,937: WARNING/MainProcess] /home/matador/anaconda3/envs/venv1/lib/python3.7/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
leak, never use this setting in production environments!
leak, never use this setting in production environments!''')
[2020-02-21 12:39:04,937: INFO/MainProcess] celery@matador-B450M-DS3H ready.
如您所见，当我执行 运行 这个命令时，我想添加到队列中的任务没有出现。其他好像都还好。
编辑：
看来这个问题的解决办法是用 celery beat 代替普通的 celery。
您可能需要 运行 一个 worker celery 进程。
celery -A proj worker -l debug
我在 Django 中设置 celery 模块时遇到了一些问题。我不仅对设置要求很困惑，而且我也很难理解我的错误。
我想在每天的特定时间运行执行一个函数，这是我以前用 crontab 做的。 Celery 显然更适合这种类型的任务，这就是为什么我试图转向这个解决方案。
设置似乎没问题，当我调用它时我看到了 celery 调度程序 运行，但是当间隔开始时我的任务没有被处理。
我看到了很多（!!!）芹菜的不同设置，所以我的代码可能包含冗余和不必要的行。
我要触发的函数称为 team_accountabillity() 并且存在于我的应用程序中 "myteam.task.py"
我有 Celery 4.4.0，我使用 rabbitMQ 作为队列管理器（我还安装了 redis，因为很多教程都用这个）。
这是我的 proj/settings.py
from __future__ import absolute_import
import os
from datetime import timedelta
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_BEAT_SCHEDULER = {
'team accountability': {
'task':'team_accountability', 'schedule': timedelta(seconds=10),
},
}
CELERY_TIMEZONE = 'UTC'
CELERY_ENABLE_UTC = True
INSTALLED_APPS = [
'celerybeat_status',
'django_celery_beat',
'myteam.apps.MyteamConfig',
]
proj/celery.py 文件：
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from celery.schedules import crontab
from django.apps import apps
from django.conf import settings
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
app = Celery('proj')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks(lambda: [n.name for n in apps.get_app_configs()])
##documentation regarding the next line:
##https://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#beat-entries
##I find it very weird that this command is very similar to the CELERY_BEAT_SCHEDULER in settings.py
##I honestly don't know what this does
@app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
sender.add_periodic_task(10.0, team_accountability, name='team accountability')
@app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
proj/init.py文件:
from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app
__all__ = ('celery_app')
最后，这是我的 app/task.py (myteam/task.py)
from __future__ import absolute_import, unicode_literals
from celery import shared_task
from myteam.models import Team
##decorator should be @shared_task or @task???
@shared_task
def team_accountability():
t_all = Team.objects.all()
for t in t_all:
total = 0
s_all = t.stocks.all()
for s in s_all:
symbol = s.symbol
queryset = Stock.objects.filter(symbol__contains=symbol).order_by("-id").first()
total = total + queryset.change_percent
t.total = round(total, 3)
t.save()
综上所述，这是我 运行 命令时得到的结果：
celery -A proj worker -l debug
我得到的是这样的：
[2020-02-21 12:39:03,420: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2020-02-21 12:39:03,420: DEBUG/MainProcess] | Worker: Building graph...
[2020-02-21 12:39:03,421: DEBUG/MainProcess] | Worker: New boot order: {StateDB, Beat, Timer, Hub, Pool, Autoscaler, Consumer}
[2020-02-21 12:39:03,429: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2020-02-21 12:39:03,429: DEBUG/MainProcess] | Consumer: Building graph...
[2020-02-21 12:39:03,447: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Gossip, Tasks, Control, Agent, Heart, event loop}
-------------- celery@matador-B450M-DS3H v4.4.0 (cliffs)
--- ***** -----
-- ******* ---- Linux-4.15.0-70-generic-x86_64-with-debian-buster-sid 2020-02-21 12:39:03
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: TheRodeoProject:0x7f00682ecc10
- ** ---------- .> transport: redis://localhost:6379//
- ** ---------- .> results: redis://localhost:6379/
- *** --- * --- .> concurrency: 12 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. TheRodeoProject.celery.debug_task
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
[2020-02-21 12:39:03,455: DEBUG/MainProcess] | Worker: Starting Hub
[2020-02-21 12:39:03,455: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:03,455: DEBUG/MainProcess] | Worker: Starting Pool
[2020-02-21 12:39:03,891: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:03,891: DEBUG/MainProcess] | Worker: Starting Consumer
[2020-02-21 12:39:03,891: DEBUG/MainProcess] | Consumer: Starting Connection
[2020-02-21 12:39:03,900: INFO/MainProcess] Connected to redis://localhost:6379//
[2020-02-21 12:39:03,901: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:03,901: DEBUG/MainProcess] | Consumer: Starting Events
[2020-02-21 12:39:03,908: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:03,908: DEBUG/MainProcess] | Consumer: Starting Mingle
[2020-02-21 12:39:03,908: INFO/MainProcess] mingle: searching for neighbors
[2020-02-21 12:39:04,924: INFO/MainProcess] mingle: all alone
[2020-02-21 12:39:04,924: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:04,925: DEBUG/MainProcess] | Consumer: Starting Gossip
[2020-02-21 12:39:04,930: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:04,930: DEBUG/MainProcess] | Consumer: Starting Tasks
[2020-02-21 12:39:04,934: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:04,934: DEBUG/MainProcess] | Consumer: Starting Control
[2020-02-21 12:39:04,935: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:04,935: DEBUG/MainProcess] | Consumer: Starting Heart
[2020-02-21 12:39:04,936: DEBUG/MainProcess] ^-- substep ok
[2020-02-21 12:39:04,936: DEBUG/MainProcess] | Consumer: Starting event loop
[2020-02-21 12:39:04,937: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2020-02-21 12:39:04,937: WARNING/MainProcess] /home/matador/anaconda3/envs/venv1/lib/python3.7/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
leak, never use this setting in production environments!
leak, never use this setting in production environments!''')
[2020-02-21 12:39:04,937: INFO/MainProcess] celery@matador-B450M-DS3H ready.
如您所见，当我执行 运行 这个命令时，我想添加到队列中的任务没有出现。其他好像都还好。
编辑：
看来这个问题的解决办法是用 celery beat 代替普通的 celery。
您可能需要 运行 一个 worker celery 进程。
celery -A proj worker -l debug