运行 ECS 任务上的 celery worker 并使用 SQS 作为代理

Question

我正在构建一个 Web 应用程序，它需要一些运行长的任务才能在 AWS ECS 上使用 celery 作为分布式任务队列。我面临的问题是我在 ECS 上的 celery worker 运行没有从 SQS 接收任务，即使它似乎已连接到它。

以下是 ECS 任务的日志。

/usr/local/lib/python3.8/site-packages/celery/platforms.py:797: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
  warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
 
 -------------- celery@ip-xxx-xxx-xxx-xxx.us-east-2.compute.internal v5.0.1 (singularity)
--- ***** ----- 
-- ******* ---- Linux-4.14.252-195.483.amzn2.x86_64-x86_64-with-glibc2.2.5 2021-12-14 06:39:58
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         emptive_portal:0x7fbfda752310
- ** ---------- .> transport:   sqs://XXXXXXXXXXXXXXXX:**@localhost//
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> emptive-celery2.fifo exchange=sync(direct) key=emptive-celery.fifo
                
[tasks]
 
  . import_export_celery.tasks.run_export_job
  . import_export_celery.tasks.run_import_job
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
2021-12-14 06:39:58 [INFO] Connected to sqs://XXXXXXXXXXXXXXXXX:**@localhost//
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
2021-12-14 06:39:58 [INFO] celery@ip-xxx-xxx-xxx-xxx.us-east-2.compute.internal ready.

需要注意的是，我有运行同一个容器，我已将其部署到 ECS，本地位于与发送任务的 django 网络服务器相同的机器上。那个celery worker 接任务没问题

我也尝试过授予 ecsTaskExecutionRole 对 SQS 的完全权限，但这似乎没有任何影响。任何帮助将不胜感激。

编辑：忘记在 django 中显示我的 celery 代理配置 settings.py

AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
    AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
    # SQS CONFIG
    CELERY_BROKER_URL = "sqs://%s:%s@" % (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
    CELERY_ACCEPT_CONTENT = ['application/json']
    CELERY_RESULT_SERIALIZER = 'json'
    CELERY_TASK_SERIALIZER = 'json'
    BROKER_TRANSPORT_OPTIONS = {
        'region': 'us-east-2',
        'polling_interval': 20,
    }
    CELERY_RESULT_BACKEND = None
    CELERY_ENABLE_REMOTE_CONTROL = False
    CELERY_SEND_EVENTS = False

Answer 1

所以我终于解决了这个问题。这个问题对我来说真的很愚蠢。 :) 我只需要在芹菜配置中用 CELERY_BROKER_TRANSPORT_OPTIONS 替换 BROKER_TRANSPORT_OPTIONS。

新配置：

AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
    AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
    # SQS CONFIG
    CELERY_BROKER_URL = "sqs://%s:%s@" % (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
    CELERY_ACCEPT_CONTENT = ['application/json']
    CELERY_RESULT_SERIALIZER = 'json'
    CELERY_TASK_SERIALIZER = 'json'
    CELERY_BROKER_TRANSPORT_OPTIONS = {
        'region': 'us-east-2',
        'polling_interval': 20,
    }
    CELERY_RESULT_BACKEND = None
    CELERY_ENABLE_REMOTE_CONTROL = False
    CELERY_SEND_EVENTS = False

运行 ECS 任务上的 celery worker 并使用 SQS 作为代理

Running celery worker on ECS Task and using SQS as a broker

django

celery

amazon-ecs

amazon-sqs