1

I am trying to schedule tasks in airflow but apparently it's not so obvious.

The task I want to run is pretty simple, here is the DAG configuration:

from datetime import datetime
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator

def print_hello():
    return 'Hello world!'

dag = DAG('hello_world', description='Simple tutorial DAG',
          schedule_interval='30 15 * * *',
          start_date=datetime(2018, 10, 3), catchup=False)

dummy_operator = DummyOperator(task_id='dummy_task', retries=3, 
dag=dag)

hello_operator = PythonOperator(task_id='hello_task', 
python_callable=print_hello, dag=dag)

dummy_operator >> hello_operator

It means I want to run it every day at 15:30. If I put this task ON today, for instance, it doesn't trigger at 15:30, but will trigger tomorrow at this time. This is related to the idea of left-border of the interval of execution (quite a strange feature of Airflow).

So the question is:

How to trigger the task exactly today at 15:30 and not tomorrow at the end of the execution interval?

Any suggestions or clarifications would be appreciated.

2 Answers 2

0

I think I got the idea.

According to [1], atomic execution series are created by Airflow. Every execution series has Execution Date which signify the beginning of the execution interval.

So in my case Execution Date (10-04T15:30:00+00:00) means the interval started 04th of October and ended 5th of October at the same time (now), executing today's task.

0

I created exactly the same DAG script today (2018-10-05T17:54:00+0). Airflow (v1.10) created a DAG run for 2018-10-04T15:30:00+0 (which should be executed at some time after 2018-10-05T15:30:00+0 [0]) and started it immediately (because it was already after 2018-10-05T15:30:00+0). It did not create a DAG run for 2018-10-03T15:30:00+0 because of catchup=False, which instructs the scheduler to only create a DAG Run for the most current instance of the DAG interval series [1].

So, the DAG run for today (2018-10-05) is the DAG run scheduled for 2018-10-04T15:30:00+0, because its scheduling interval ends today.

The reason why you do not see a DAG run for today (i.e., the one for 2018-10-04T15:30:00+0) in the Airflow Web UI in your case may be that you first set start_date to today, scheduled the DAG, and after that tried to change the start_date to an earlier day. The scheduler actually created a DAG run for an earlier date, but the Web UI just didn't display it. The easiest way to fix this is to change the dag_id (e.g., to hello_world_2 etc.) when you change the start_date or schedule_interval [2] and restart the scheduler.

6
  • SergiyKolesnikov, thank you for your reply. Concerning the fact that I am not seeing the DAG for today - shouldn't restart of the airflow database help? That's what I was trying to do. Oct 5, 2018 at 20:00
  • @NikitaVostrosablin You mean airflow resetdb; airflow initdb? It is quite intrusive and will remove other DAGs too, but, yes, that should help too. Oct 5, 2018 at 20:07
  • Unfortunately, this didn't help. I created a new DAG script (2018-10-05T22:22:00+0), restarted the database together with the webserver and the scheduler, putting catchup=True. This created two DAG runs for (10-03T15:30:00+00:00) and (10-04T15:30:00+00:00). No run for today :( Oct 5, 2018 at 20:27
  • SergiyKolesnikov, According to the DAG interval series documentation, the atomic execution series are created by Airflow, and the current day can't appear until the interval will be complete (meaning tomorrow at the same time). So that's the problem - how do I run task if I need it exactly today? Oct 5, 2018 at 20:45
  • @NikitaVostrosablin Actually, the DAG run for 10-04T15:30:00+00:00 will be executed at some time after 10-05T15:30:00+00:00. So this DAG run is actually for today, even though it is scheduled for yesterday. This is the most weird thing about Airflow, but this is how it works. Oct 5, 2018 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.