如何给芹菜中的每个节点命名

Question

我们在烧瓶应用程序中使用芹菜。我们的各种任务有很多队列，我们使用 supervisord 来运行个队列。我们使用cloudamqp 经纪人。

主管配置示例如下：

[program:my-queue]
command=/home/ubuntu/opt/proect/venv/bin/celery -A async_runner worker -Q my_queue --loglevel=INFO --without-gossip --without-mingle --autoscale=1,1  -c 1
environment=PYTHONPATH=/home/ubuntu/opt/project/,PRODUCTION_ENVIRONMENT=true
directory=/home/ubuntu/opt/project/app

process_name = %(program_name)s_%(process_num)02d
user=ubuntu
numprocs=2
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600
priority=998

我们收到以下错误，队列停止。

[2017-04-06 12:43:06,759: WARNING/MainProcess] /home/ubuntu/opt/project/venv/local/lib/python2.7/site-packages/kombu/pidbox.py:75: UserWarning: A node named celery@ip-xxx-yy-yy-yy is already using this process mailbox!

Maybe you forgot to shutdown the other node or did not do so properly?
Or if you meant to start multiple nodes on the same host please make sure
you give each node a unique node name!

  warnings.warn(W_PIDBOX_IN_USE.format(node=self))

问题：如何给每个节点命名？

当我运行celery -A async_runner status时给5个节点在线留言

celery@ip-ip_value_here-: OK
celery@ip-ip_value_here-: OK
celery@ip-ip_value_here-: OK
celery@ip-ip_value_here-: OK
celery@ip-ip_value_here-: OK

5 nodes online.

Answer 1

下面是我们如何运行为公司的每个队列配备一名芹菜工人。入口点包含：

echo QUEUES: ${QUEUES}  # comma separated list of queue names
export NUM_QUEUES=$(python -c "print len('$QUEUES'.split(','))")
echo NUM_QUEUES: ${NUM_QUEUES}

supervisord &

主管配置文件有这个：

[program:worker]
command=/path/to/celery_worker.sh %(process_num)s
# supervisor uses the special ENV_ prefix to get the environment variable set above
numprocs=%(ENV_NUM_QUEUES)s

最后 celery_worker.sh 包含如下内容：

QUEUE=$(python -c "print '$QUEUES'.split(',')[]")  #  = process_num above
celery worker -n "worker.${QUEUE}" -Q ${QUEUE}  # -n sets the name of the worker

Answer 2

使用选项 -n SOMENODENAME@%%h。 @%%h 将确保每个工作人员都有一个唯一的节点名

如何给芹菜中的每个节点命名

How to give name to each node in celery

python

celery

supervisord