如何用气流声明复杂的依赖关系

Question

我想用这种方式声明依赖关系，但它不起作用

start >> [a, b >> [b1, b2], c] >> end

它引发了这个错误

log: Relationships can only be set between Operators; received list

我们怎样才能做到这一点？

Answer 1

尝试使用 TaskGroup:

from airflow.models import DAG
from airflow.operators.dummy import DummyOperator
from airflow.utils.task_group import TaskGroup

dag = DAG(
    dag_id="test",
    start_date=datetime(2021, 6, 30),
    schedule_interval=None,
    catchup=False,
)

with dag:
    start = DummyOperator(task_id="start")
    end = DummyOperator(task_id="end")

    with TaskGroup("task_group") as task_group:
        a = DummyOperator(task_id="a")
        b = DummyOperator(task_id="b")
        b1 = DummyOperator(task_id="b1")
        b2 = DummyOperator(task_id="b2")
        c = DummyOperator(task_id="c")

        b >> [b1, b2]


    start >> task_group >> end

如何用气流声明复杂的依赖关系

How to declare complex dependencies with airflow

airflow