芹菜工人收到任务但没有执行
Celery worker received task but not executing
我曾尝试在 windows 10
上使用 celery=4.2.0
、Python=3.6
和 Rabbitmq=3.7.9
执行简单的任务,该任务已被工作人员接收但没有执行并且保持不变。
任务减量:
@app.task()
def simpleTask(*args):
logging.info('{}'.format(args[0]))
任务执行命令:
simpleTask.s('1').apply_async()
我尝试使用 -P gevent(version=1.4.0)
& eventlet(version=0.24.1)
运行 应用程序,但没有成功。尽管使用 solo 选项时任务执行正确,但对于我的最终目标 运行 单线程环境中的工作人员执行的任务不符合我的要求。
芹菜工人命令:
argv = [
'worker',
'-l=DEBUG',
'-P=eventlet',
'--without-gossip',
'--without-mingle',
'--without-heartbeat',
'-Ofair',
]
app.worker_main(argv)
输出:
[2019-03-10 23:08:18,612: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2019-03-10 23:08:18,618: DEBUG/MainProcess] | Worker: Building graph...
[2019-03-10 23:08:18,619: DEBUG/MainProcess] | Worker: New boot order: {StateDB, Timer, Hub, Pool, Autoscaler, Beat, Consumer}
[2019-03-10 23:08:18,938: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2019-03-10 23:08:18,939: DEBUG/MainProcess] | Consumer: Building graph...
[2019-03-10 23:08:18,951: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Agent, Events, Heart, Mingle, Gossip, Tasks, Control, event loop}
[2019-03-10 23:08:18,962: DEBUG/MainProcess] | Worker: Starting Pool
-------------- celery@DESKTOP-IEMN7E0 v4.2.1 (windowlicker)
[2019-03-10 23:08:18,962: DEBUG/MainProcess] ^-- substep ok
---- **** -----
[2019-03-10 23:08:18,962: DEBUG/MainProcess] | Worker: Starting Consumer
--- * *** * -- Windows-10-10.0.17134-SP0 2019-03-10 23:08:18
[2019-03-10 23:08:18,962: DEBUG/MainProcess] | Consumer: Starting Connection
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: workflowEngine.Main.Server.tasks:0x2771cf629e8
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
- ** ---------- .> results: mongodb://
- *** --- * --- .> concurrency: 4 (eventlet)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. PanelTask
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. workflowEngine.Main.Server.tasks.finalChordTask
. workflowEngine.Main.Server.tasks.simpleTask
[2019-03-10 23:08:18,978: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@DESKTOP-IEMN7E0', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.1', 'product': 'RabbitMQ', 'version': '3.7.9'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-03-10 23:08:18,980: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2019-03-10 23:08:18,981: DEBUG/MainProcess] ^-- substep ok
[2019-03-10 23:08:18,981: DEBUG/MainProcess] | Consumer: Starting Events
[2019-03-10 23:08:18,998: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@DESKTOP-IEMN7E0', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.1', 'product': 'RabbitMQ', 'version': '3.7.9'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-03-10 23:08:19,001: DEBUG/MainProcess] ^-- substep ok
[2019-03-10 23:08:19,001: DEBUG/MainProcess] | Consumer: Starting Tasks
[2019-03-10 23:08:19,010: DEBUG/MainProcess] using channel_id: 1
[2019-03-10 23:08:19,028: DEBUG/MainProcess] Channel open
[2019-03-10 23:08:19,037: DEBUG/MainProcess] ^-- substep ok
[2019-03-10 23:08:19,037: DEBUG/MainProcess] | Consumer: Starting Control
[2019-03-10 23:08:19,052: DEBUG/MainProcess] ^-- substep ok
[2019-03-10 23:08:19,052: DEBUG/MainProcess] | Consumer: Starting event loop
[2019-03-10 23:08:19,054: INFO/MainProcess] celery@DESKTOP-IEMN7E0 ready.
[2019-03-10 23:08:19,055: DEBUG/MainProcess] basic.qos: prefetch_count->16
[2019-03-10 23:09:48,701: INFO/MainProcess] Received task: workflowEngine.Main.Server.tasks.simpleTask[13644976-55e4-4df4-a48a-a9274ff3daff]
[2019-03-10 23:09:48,703: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x000002771E1B9C80> (args:('workflowEngine.Main.Server.tasks.simpleTask', '13644976-55e4-4df4-a48a-a9274ff3daff', {'lang': 'py', 'task': 'workflowEngine.Main.Server.tasks.simpleTask', 'id': '13644976-55e4-4df4-a48a-a9274ff3daff', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '13644976-55e4-4df4-a48a-a9274ff3daff', 'parent_id': None, 'argsrepr': "('1',)", 'kwargsrepr': '{}', 'origin': 'gen16440@DESKTOP-IEMN7E0', 'reply_to': 'e1566cdc-a4b7-3017-8167-36ea0a95af14', 'correlation_id': '13644976-55e4-4df4-a48a-a9274ff3daff', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[["1"], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
经过一些调试后,我发现我的问题是错误地传递了 -P 参数,令人惊讶的是,根据 celery 输出,应用程序确实识别出我想使用 eventlet
包名字.
argv = [
'worker',
'-l=DEBUG',
'-P',
'eventlet',
'-c=100',
'--without-gossip',
'--without-mingle',
'--without-heartbeat',
'-Ofair',
]
app.worker_main(argv)
对于 eventlet
和 gevent
,此命令对我来说效果很好
我曾尝试在 windows 10
上使用 celery=4.2.0
、Python=3.6
和 Rabbitmq=3.7.9
执行简单的任务,该任务已被工作人员接收但没有执行并且保持不变。
任务减量:
@app.task()
def simpleTask(*args):
logging.info('{}'.format(args[0]))
任务执行命令:
simpleTask.s('1').apply_async()
我尝试使用 -P gevent(version=1.4.0)
& eventlet(version=0.24.1)
运行 应用程序,但没有成功。尽管使用 solo 选项时任务执行正确,但对于我的最终目标 运行 单线程环境中的工作人员执行的任务不符合我的要求。
芹菜工人命令:
argv = [
'worker',
'-l=DEBUG',
'-P=eventlet',
'--without-gossip',
'--without-mingle',
'--without-heartbeat',
'-Ofair',
]
app.worker_main(argv)
输出:
[2019-03-10 23:08:18,612: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2019-03-10 23:08:18,618: DEBUG/MainProcess] | Worker: Building graph...
[2019-03-10 23:08:18,619: DEBUG/MainProcess] | Worker: New boot order: {StateDB, Timer, Hub, Pool, Autoscaler, Beat, Consumer}
[2019-03-10 23:08:18,938: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2019-03-10 23:08:18,939: DEBUG/MainProcess] | Consumer: Building graph...
[2019-03-10 23:08:18,951: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Agent, Events, Heart, Mingle, Gossip, Tasks, Control, event loop}
[2019-03-10 23:08:18,962: DEBUG/MainProcess] | Worker: Starting Pool
-------------- celery@DESKTOP-IEMN7E0 v4.2.1 (windowlicker)
[2019-03-10 23:08:18,962: DEBUG/MainProcess] ^-- substep ok
---- **** -----
[2019-03-10 23:08:18,962: DEBUG/MainProcess] | Worker: Starting Consumer
--- * *** * -- Windows-10-10.0.17134-SP0 2019-03-10 23:08:18
[2019-03-10 23:08:18,962: DEBUG/MainProcess] | Consumer: Starting Connection
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: workflowEngine.Main.Server.tasks:0x2771cf629e8
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
- ** ---------- .> results: mongodb://
- *** --- * --- .> concurrency: 4 (eventlet)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. PanelTask
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. workflowEngine.Main.Server.tasks.finalChordTask
. workflowEngine.Main.Server.tasks.simpleTask
[2019-03-10 23:08:18,978: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@DESKTOP-IEMN7E0', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.1', 'product': 'RabbitMQ', 'version': '3.7.9'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-03-10 23:08:18,980: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2019-03-10 23:08:18,981: DEBUG/MainProcess] ^-- substep ok
[2019-03-10 23:08:18,981: DEBUG/MainProcess] | Consumer: Starting Events
[2019-03-10 23:08:18,998: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@DESKTOP-IEMN7E0', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.1', 'product': 'RabbitMQ', 'version': '3.7.9'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-03-10 23:08:19,001: DEBUG/MainProcess] ^-- substep ok
[2019-03-10 23:08:19,001: DEBUG/MainProcess] | Consumer: Starting Tasks
[2019-03-10 23:08:19,010: DEBUG/MainProcess] using channel_id: 1
[2019-03-10 23:08:19,028: DEBUG/MainProcess] Channel open
[2019-03-10 23:08:19,037: DEBUG/MainProcess] ^-- substep ok
[2019-03-10 23:08:19,037: DEBUG/MainProcess] | Consumer: Starting Control
[2019-03-10 23:08:19,052: DEBUG/MainProcess] ^-- substep ok
[2019-03-10 23:08:19,052: DEBUG/MainProcess] | Consumer: Starting event loop
[2019-03-10 23:08:19,054: INFO/MainProcess] celery@DESKTOP-IEMN7E0 ready.
[2019-03-10 23:08:19,055: DEBUG/MainProcess] basic.qos: prefetch_count->16
[2019-03-10 23:09:48,701: INFO/MainProcess] Received task: workflowEngine.Main.Server.tasks.simpleTask[13644976-55e4-4df4-a48a-a9274ff3daff]
[2019-03-10 23:09:48,703: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x000002771E1B9C80> (args:('workflowEngine.Main.Server.tasks.simpleTask', '13644976-55e4-4df4-a48a-a9274ff3daff', {'lang': 'py', 'task': 'workflowEngine.Main.Server.tasks.simpleTask', 'id': '13644976-55e4-4df4-a48a-a9274ff3daff', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '13644976-55e4-4df4-a48a-a9274ff3daff', 'parent_id': None, 'argsrepr': "('1',)", 'kwargsrepr': '{}', 'origin': 'gen16440@DESKTOP-IEMN7E0', 'reply_to': 'e1566cdc-a4b7-3017-8167-36ea0a95af14', 'correlation_id': '13644976-55e4-4df4-a48a-a9274ff3daff', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[["1"], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
经过一些调试后,我发现我的问题是错误地传递了 -P 参数,令人惊讶的是,根据 celery 输出,应用程序确实识别出我想使用 eventlet
包名字.
argv = [
'worker',
'-l=DEBUG',
'-P',
'eventlet',
'-c=100',
'--without-gossip',
'--without-mingle',
'--without-heartbeat',
'-Ofair',
]
app.worker_main(argv)
对于 eventlet
和 gevent