0

Suppose I have following DAG dependencies:

        /> DAG B -->....
       /
      /
DAG A ---> DAG C -->...
      \
       \
        \> DAG D -->...

These DAGs are running daily, and output of DAG A is consumed by DAG B,C,D respectively.

If the DAG A failed (e.g. Spark job had a bug and we want to re-execute it after the fix), how to make sure we also re-run DAGs B,C,D etc so that now they run with fixed DAG A output ? Can it be done automatically ?

2 Answers 2

1

You could have a separate Main DAG (controller_dag) and use TriggerDagRunOperator to trigger different DAGs.

The dependencies between tasks using TriggerDagRunOperator can be set in the following way:

dag_a >> [dag_b, dag_c, dag_d] >> ...
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.