Heroku + Celery：关键工作人员超时 (503)

Question

我在 Heroku 上部署了一个 Flask 应用程序，我正在使用 Celery+Redis 进行异步处理。发送请求时测试页面未加载，我收到此错误：

2021-10-17T13:32:59.278920+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/test_route" host=www.mysite.dev request_id=f387a680-dca6-4e20-8f97-7f4d4a45d17b fwd="73.252.142.129,108.162.215.101" dyno=web.1 connect=0ms service=30000ms status=503 bytes=0 protocol=https
2021-10-17T13:32:59.776324+00:00 app[web.1]: [2021-10-17 13:32:59 +0000] [19] [CRITICAL] WORKER TIMEOUT (pid:23)
2021-10-17T13:32:59.778326+00:00 app[web.1]: [2021-10-17 13:32:59 +0000] [23] [INFO] Worker exiting (pid: 23)

我的测试路线：

@main_bp.route('/test_route')
def test_route():
    celery_tasks.test_task.apply_async()

    return render_template('main/test.html')

芹菜任务：

@celery_app.task()
def test_task():
    print('Test task succeeded')

    return 1

工作人员在任务被解雇前看起来很好：

2021-10-17T13:54:28.080039+00:00 app[worker.1]: [2021-10-17 13:54:28,079: INFO/MainProcess] mingle: all alone
2021-10-17T13:54:28.080111+00:00 app[worker.1]: [2021-10-17 13:54:28,080: DEBUG/MainProcess] ^-- substep ok
2021-10-17T13:54:28.080178+00:00 app[worker.1]: [2021-10-17 13:54:28,080: DEBUG/MainProcess] | Consumer: Starting Tasks
2021-10-17T13:54:28.086858+00:00 app[worker.1]: [2021-10-17 13:54:28,086: DEBUG/MainProcess] ^-- substep ok
2021-10-17T13:54:28.086956+00:00 app[worker.1]: [2021-10-17 13:54:28,086: DEBUG/MainProcess] | Consumer: Starting Control
2021-10-17T13:54:28.094411+00:00 app[worker.1]: [2021-10-17 13:54:28,094: DEBUG/MainProcess] ^-- substep ok
2021-10-17T13:54:28.094481+00:00 app[worker.1]: [2021-10-17 13:54:28,094: DEBUG/MainProcess] | Consumer: Starting Gossip
2021-10-17T13:54:28.100571+00:00 app[worker.1]: [2021-10-17 13:54:28,100: DEBUG/MainProcess] ^-- substep ok
2021-10-17T13:54:28.100633+00:00 app[worker.1]: [2021-10-17 13:54:28,100: DEBUG/MainProcess] | Consumer: Starting event loop
2021-10-17T13:54:28.100745+00:00 app[worker.1]: [2021-10-17 13:54:28,100: DEBUG/MainProcess] | Worker: Hub.register Pool...
2021-10-17T13:54:28.101184+00:00 app[worker.1]: [2021-10-17 13:54:28,101: INFO/MainProcess] celery@82da4ee3-bdcc-46c6-b65b-d6221ba1693e ready.
2021-10-17T13:54:28.101315+00:00 app[worker.1]: [2021-10-17 13:54:28,101: DEBUG/MainProcess] basic.qos: prefetch_count->32

我从路由中删除了 celery 任务 (.apply_async) 并确认页面在没有它的情况下加载
在本地，我将 time.sleep(30) 添加到 celery 任务以确认该任务已卸载到 worker，并且在任务仍在运行时加载页面。这按预期工作

我不确定为什么页面加载在生产中停滞，因为异步处理的目的是不阻止请求。我也不确定如何深入挖掘并调试此错误 - 在文档中找不到太多内容。

我在文档中找到了这个：Debugging Request Timeouts，但它只指向长运行任务和无限循环，这里都不是

Answer 1

这已通过更改 celery_tasks.py 中使用的 celery 应用程序解决。

这不起作用：

#celery_tasks.py

from celery import Celery

celery_app = Celery()

@celery_app.task()
def test_task():
    ...

这个有效：

#celery_tasks.py

from extensions import celery

celery_app = celery

@celery_app.task()
def test_task():
    ...

其中：

#extensions.py
import flask
from celery import Celery

class FlaskCelery(Celery):
    def __init__(self, *args, **kwargs):

        super(FlaskCelery, self).__init__(*args, **kwargs)
        self.patch_task()

        if 'app' in kwargs:
            self.init_app(kwargs['app'])

    def patch_task(self):
        TaskBase = self.Task
        _celery = self

        class ContextTask(TaskBase):
            abstract = True

            def __call__(self, *args, **kwargs):
                if flask.has_app_context():
                    return TaskBase.__call__(self, *args, **kwargs)
                else:
                    with _celery.app.app_context():
                        return TaskBase.__call__(self, *args, **kwargs)

        self.Task = ContextTask

    def init_app(self, app):
        self.app = app
        self.config_from_object(app.config)


celery = FlaskCelery()

和__init__.py使用相同的class

Heroku + Celery：关键工作人员超时 (503)

Heroku + Celery: Critical Worker Timeout (503)

python

flask

heroku

redis

celery