如何在气流上的特定 dag 中同时限制任务数 运行?
how to limit the number of tasks run simultaneously in a specific dag on airflow?
我在 air-flow
中有一个 Dag
看起来像这样:
start >> [task1,task2,....task16] >> end
我想限制运行同时在这个dag中的任务为4个。
我知道有 'max_active_tasks_per_dag'
参数,但它会影响所有 dag,在我的情况下,我只想为我的 dag 定义它。
我该怎么做?甚至有可能吗?
如果你查看参数的描述,你会发现你可以为每个 dag 配置它:
max_active_tasks_per_dag New in version 2.2.0.
The maximum number of task instances allowed to run concurrently in
each DAG. To calculate the number of tasks that is running
concurrently for a DAG, add up the number of running tasks for all DAG
runs of the DAG. This is configurable at the DAG level with
max_active_tasks, which is defaulted as max_active_tasks_per_dag.
我在 air-flow
中有一个 Dag
看起来像这样:
start >> [task1,task2,....task16] >> end
我想限制运行同时在这个dag中的任务为4个。
我知道有 'max_active_tasks_per_dag'
参数,但它会影响所有 dag,在我的情况下,我只想为我的 dag 定义它。
我该怎么做?甚至有可能吗?
如果你查看参数的描述,你会发现你可以为每个 dag 配置它:
max_active_tasks_per_dag New in version 2.2.0.
The maximum number of task instances allowed to run concurrently in each DAG. To calculate the number of tasks that is running concurrently for a DAG, add up the number of running tasks for all DAG runs of the DAG. This is configurable at the DAG level with max_active_tasks, which is defaulted as max_active_tasks_per_dag.