16

I am using Airflow 1.7.1.3 installed using pip

I would like to limit the logging to ERROR level for the workflow being executed by the scheduler. Could not find anything beyond setting log files location in the settings.py file.

Also the online resources led me to this google group discussion here but not much info here as well

Any idea how to control logging in Airflow?

5 Answers 5

16

I tried below work around and it seems to be working to set LOGGING_LEVEL outside of settings.py:

  • Update settings.py:

    • Remove or comment line:
      LOGGING_LEVEL = logging.INFO

    • Add line:
      LOGGING_LEVEL = os.path.expanduser(conf.get('core', 'LOGGING_LEVEL'))

  • Update airflow.cfg configuration file:

    • Add line under [core]: logging_level = WARN

    • Restart webserver and scheduler services

  • Use environment vaiable AIRFLOW__CORE__LOGGING_LEVEL=WARN.

See the official docs for details.

2
  • LOGGING_LEVEL = conf.get('core', 'LOGGING_LEVEL') why are you expanding the ~ ? This setting is not a path.
    – ozw1z5rd
    Commented Mar 13, 2018 at 11:21
  • 2
    Note that AIRFLOW__CORE__LOGGING_LEVEL has been deprecated in favour of AIRFLOW__LOGGING__LOGGING_LEVEL since 2.0.0: airflow.apache.org/docs/apache-airflow/stable/…
    – Alexander
    Commented Sep 27, 2022 at 16:44
8

The logging functionality and its configuration will be changed in version 1.9 with this commit

1
  • 1
    Thanks for the update. Marking this as accepted answer as upgrading airflow is a better solution Commented Oct 20, 2017 at 10:08
3

As @Dimo Boyadzhiev pointed the change, Adding the more info path for documentaion.

File - $AIRFLOW_HOME/airflow.cfg

# Logging level
logging_level = INFO
fab_logging_level = WARN
1

Only solution I am aware of is changing LOGGING_LEVEL in settings.py file. Default level is set to INFO.

AIRFLOW_HOME = os.path.expanduser(conf.get('core', 'AIRFLOW_HOME'))
SQL_ALCHEMY_CONN = conf.get('core', 'SQL_ALCHEMY_CONN') 
LOGGING_LEVEL = logging.INFO
DAGS_FOLDER = os.path.expanduser(conf.get('core', 'DAGS_FOLDER'))
0
0

If you are using docker-compose.yaml

x-airflow-common:
  &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.2}
  environment:
    &airflow-common-env
    #the other parameters 
    AIRFLOW__CORE__LOGGING_LEVEL: DEBUG #add 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.