芹菜昆布在 self.connections.acquire 后失败

Question

当我的芹菜服务运行 7-10 天后，我无缘无故地收到了这个异常，这导致我的任务无法处理。重新启动 celery 可以解决问题。

INTERNAL ERROR: RuntimeError('Acquire on closed pool',)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 253, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 201, in on_error
    R = I.handle_error_state(task, eager=eager)
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 85, in handle_error_state
    }[self.state](task, store_errors=store_errors)
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 118, in handle_failure
    req.id, exc, einfo.traceback, request=req,
  File "/usr/lib/python2.7/dist-packages/celery/backends/base.py", line 121, in mark_as_failure
    traceback=traceback, request=request)
  File "/usr/lib/python2.7/dist-packages/celery/backends/amqp.py", line 124, in store_result
    with self.app.amqp.producer_pool.acquire(block=True) as producer:
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 868, in acquire
    R = self.prepare(R)
  File "/usr/lib/python2.7/dist-packages/kombu/pools.py", line 63, in prepare
    conn = self._acquire_connection()
  File "/usr/lib/python2.7/dist-packages/kombu/pools.py", line 38, in _acquire_connection
    return self.connections.acquire(block=True)
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 859, in acquire
    raise RuntimeError('Acquire on closed pool')
RuntimeError: Acquire on closed pool

软件版本

software -> celery:3.1.20 (Cipater) kombu:3.0.35 py:2.7.6
            billiard:3.3.0.22 py-amqp:1.4.9
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:amqp

CELERY_ACCEPT_CONTENT: ['json', 'pickle', 'yaml']
CELERY_ENABLE_UTC: True
CELERY_IGNORE_RESULT: False
CELERY_IMPORTS: 
    ('catalogue.app.voice.cluster.deploy_cluster',
 'catalogue.app.common.install_uc',
 'hypervisor.app.deploy_esx',
 'hypervisor.app.vm_operations',
 'tools.deploy_tools')
CELERYD_CHDIR: '/usr/local/src/imbue/application/app'

CELERY_TASK_RESULT_EXPIRES: 18000
CELERY_RESULT_PERSISTENT: True
CELERY_TIMEZONE: 'US/Eastern'
BROKER_URL: 'amqp://******:********@rabbitmq:5672//'
CELERY_RESULT_BACKEND: 'amqp'

现在唯一的解决方法是重新启动。 Ubuntu 14.04 2 GB RAM/2 CPU/40 GB 硬盘

Answer 1

这看起来像 bug in celery. Asksol fixed this few days back。

您可以从源代码安装芹菜并尝试一下。如果它仍然导致问题，请在 github 上创建新问题。

芹菜昆布在 self.connections.acquire 后失败

Celery kombu fails after self.connections.acquire

python

rabbitmq

celery

kombu