I was learning apache airflow and found that there is an operator called DummyOperator. I googled about its use case, but couldn't find anything that I can understand. Can anyone here please discuss its use case?
3 Answers
Operator that does literally nothing. It can be used to group tasks in a DAG.
https://airflow.apache.org/_api/airflow/operators/dummy_operator/index.html
as far as I know, at least to two case:
- test purpose. in dags, the dummy operation just between upstream and downstream, later, you can replace the true operator.
- Workflow purpose: BranchPythonOperator work with DummyOperator. If you want to skip some tasks, keep in mind that you can’t have an empty path, if so make a dummy task. https://airflow.apache.org/concepts.html#workflows
-
14adding another use-case here: if you have a set of 'd' downstream tasks that must all execute after 'u' upstream tasks, then rather than creating a 'mesh' of 'u x d' connections in between, you can introduce a
dummyoperator
in between these two sets of tasks and get away with 'u + d' connections. This certainly provides an aesthetic improvement in rendering of your DAG (looks simpler, lighter to render), but can also potentially decrease the scheduling overhead (and scheduler latency) since much lesser number of conditions have to be evaluated to check scheduling-ready state Aug 14, 2019 at 4:30 -
@Yong Wang how can I keep some latency between two tasks ...i.e. after first task successful wait 1 hr and start second task.......how to check how my first job executed with shellscript is success or not ? May 27, 2020 at 5:21
-
1@BdEngineer Thanks for question. As far as I think the most convenient mode is to create a bashoperator which sleet 1hour'. And add it between the upstream task and downstream task. In this way, the original downstream task will be delay 1 hour after original upsteam task successed.. Hope it help you. May 27, 2020 at 6:58
-
@y2k-shubham vote for the sense of aesthetic. Clean and elegant way. May 27, 2020 at 7:01
dummy_operator is used in BranchPythonOperator where we decide next task based on some condition.
For example:
-> task C->task D
task A -> task B -> task F
-> task E(Dummy)
So let's suppose we have some condition in task B which decides whether to follow [task C->task D] or task E(Dummy) to reach task F.
Since we cannot leave else condition empty we have to put dummy operator which does nothing just skip or bypass.
-
3Why can't the condition create a dependency directly from
task B
totask F
? Mar 17, 2021 at 0:28
Another use case: I've implemented a framework that returns an Operator. In most cases this is a PostgresOperator but under some user-specified configuration there's no SQL to run but the caller still expects an Operator so I return a DummyOperator rather than a PostgresOperator with trivial SQL like "select 1;".