Airflow CeleryExecutor - 'int' 对象在 Celery 中没有属性 'startswith'
Airflow CeleryExecutor - 'int' object has no attribute 'startswith' in Celery
Airflow 2.0 正在排队但未在我的开发环境中启动任务。
DAG 和 Pool 设置有效,但是当我触发它们时每个 dag 中的所有任务都是 queued
,并且永远不会 运行。
输入 airflow celery worker
时,出现以下错误:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/celery_command.py", line 188, in worker
_run_worker(options=options, skip_serve_logs=skip_serve_logs)
File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/celery_command.py", line 94, in _run_worker
celery_app.worker_main(options)
File "/usr/local/lib/python3.8/site-packages/celery/app/base.py", line 365, in worker_main
return instantiate(
File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 283, in execute_from_commandline
self.maybe_patch_concurrency(argv)
File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 315, in maybe_patch_concurrency
maybe_patch_concurrency(argv, *pool_option)
File "/usr/local/lib/python3.8/site-packages/celery/__init__.py", line 143, in maybe_patch_concurrency
pool = _find_option_with_arg(argv, short_opts, long_opts)
File "/usr/local/lib/python3.8/site-packages/celery/__init__.py", line 95, in _find_option_with_arg
if arg.startswith('-'):
AttributeError: 'int' object has no attribute 'startswith'
在我的暂存、生产环境中,运行 任务以及我检查 airflow celery worker
都没有问题。它会启动或警告我它已经 运行(如预期的那样)。
envs 之间没有区别,但我猜测问题发生在服务器上最近部署之后。
正如我所见,celery 收到了错误的参数:
AttributeError: 'int' object has no attribute 'startswith'
但是如何跟踪哪些参数 Airflow 试图传递给 celery?
我不知道如何调试它。
通过将 celery 升级到最新版本从 4.4.2
到 5.1.2
解决。
似乎版本 4.4.2
(气流依赖之一)有一个参数错误。
如果有任何其他解决此问题的建议,请随时在此处提及。
对于 airflow 2.1.0+,你需要 celery 5.1.2+
文档在这里:https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/index.html#pip-requirements 它解释了对 celery 5.1.2+
的需求
Airflow 2.0 正在排队但未在我的开发环境中启动任务。
DAG 和 Pool 设置有效,但是当我触发它们时每个 dag 中的所有任务都是 queued
,并且永远不会 运行。
输入 airflow celery worker
时,出现以下错误:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/celery_command.py", line 188, in worker
_run_worker(options=options, skip_serve_logs=skip_serve_logs)
File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/celery_command.py", line 94, in _run_worker
celery_app.worker_main(options)
File "/usr/local/lib/python3.8/site-packages/celery/app/base.py", line 365, in worker_main
return instantiate(
File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 283, in execute_from_commandline
self.maybe_patch_concurrency(argv)
File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 315, in maybe_patch_concurrency
maybe_patch_concurrency(argv, *pool_option)
File "/usr/local/lib/python3.8/site-packages/celery/__init__.py", line 143, in maybe_patch_concurrency
pool = _find_option_with_arg(argv, short_opts, long_opts)
File "/usr/local/lib/python3.8/site-packages/celery/__init__.py", line 95, in _find_option_with_arg
if arg.startswith('-'):
AttributeError: 'int' object has no attribute 'startswith'
在我的暂存、生产环境中,运行 任务以及我检查 airflow celery worker
都没有问题。它会启动或警告我它已经 运行(如预期的那样)。
envs 之间没有区别,但我猜测问题发生在服务器上最近部署之后。
正如我所见,celery 收到了错误的参数:
AttributeError: 'int' object has no attribute 'startswith'
但是如何跟踪哪些参数 Airflow 试图传递给 celery? 我不知道如何调试它。
通过将 celery 升级到最新版本从 4.4.2
到 5.1.2
解决。
似乎版本 4.4.2
(气流依赖之一)有一个参数错误。
如果有任何其他解决此问题的建议,请随时在此处提及。
对于 airflow 2.1.0+,你需要 celery 5.1.2+
文档在这里:https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/index.html#pip-requirements 它解释了对 celery 5.1.2+
的需求