36

How do you manage your application logs in AWS elastic beanstalk? Which file you write you application logs to?

I'm using the following Logging configuration in my development environment but this doesn't work when I deploy in AWS.

DEBUG_LOG_DIR = BASE_DIR + "/django_debug.log"
LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    # How to format the output
    'formatters': {
        'standard': {
            'format' : "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
            'datefmt' : "%d/%b/%Y %H:%M:%S"
        },
    },
    # Log handlers (where to go)
    'handlers': {
        'null': {
            'level':'DEBUG',
            'class':'django.utils.log.NullHandler',
        },
        'log_file': {
            'level':'DEBUG',
            'class':'logging.handlers.RotatingFileHandler',
            'filename': DEBUG_LOG_DIR,
            'maxBytes': 50000,
            'backupCount': 2,
            'formatter': 'standard',
        },
        'console':{
            'level':'INFO',
            'class':'logging.StreamHandler',
            'formatter': 'standard'
        },
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler',
        },
    },
    # Loggers (where does the log come from)
    'loggers': {
        'repackager': {
            'handlers': ['console', 'log_file'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'django': {
            'handlers':['console'],
            'propagate': True,
            'level':'WARN',
        },
        'django.db.backends': {
            'handlers': ['console', 'log_file'],
            'level': 'WARN',
            'propagate': False,
        },
        '': {
            'handlers': ['console', 'log_file'],
            'level': 'DEBUG',
        },
    }
}

9 Answers 9

41

I had a similar issue but on Elastic Beanstalk, so I created a config file (e.g. applogs.config) in .ebextensions folder of the app. This creates the app-logs folder if it is not there already and sets the file permissions and owner so that the app can write its logs there.

commands:
  00_create_dir:
    command: mkdir -p /var/log/app-logs
  01_change_permissions:
    command: chmod g+s /var/log/app-logs
  02_change_owner:
    command: chown wsgi:wsgi /var/log/app-logs

Finally, in your Django settings:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': '/var/log/app-logs/django.log',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

Aditionally, if you want your log to be accessible from beanstalk logs using the web, add this to your file in .ebextensions

files:
  "/opt/elasticbeanstalk/tasks/taillogs.d/django.conf":
    mode: "000755"
    owner: root
    group: root
    content: |
      /var/log/app-logs/django.log
13
  • 7
    in my setup id had to use another user: command: chown wsgi:wsgi /var/log/app-logs
    – linqu
    Sep 22, 2016 at 18:48
  • 4
    It is actually better to use /opt/python/log/ for the logs. The directory already exists, no need to create it, and it's packaged when getting log bundle or saving them to cloudwatch.
    – Thierry J.
    Aug 3, 2018 at 12:58
  • 6
    As suggested in another answer, I added the following to avoid one final permission error: 03_change_default_owner: command: setfacl -d -m g::rw /var/log/app-logs
    – jmwicks
    Jan 22, 2019 at 4:46
  • 7
    I beat my head for hours on why this didn't work for me. I was getting: ValueError: Unable to configure handler 'file': [Errno 13] Permission denied: '/var/log/app-logs/django.log' It turns out it was because the log file already existed, and I needed to make my chmod and chown recursive. Hope this helps someone.
    – Oded
    Jun 11, 2019 at 18:35
  • 18
    Note for Amazon Linux 2 with gunicorn, the user and group changed from wsgi:wsgi to webapp:webapp
    – Oded
    Aug 6, 2020 at 2:29
9

Ok, I figured out a way to do it.

First I connected via ssh to ec2 machine, then I create a folder in /var/log called app_logs with root user:

mkdir /var/log/app_logs

After that I did the follow:

cd /var/log/
chmod g+s app_logs/
setfacl -d -m g::rw app_logs/
chown wsgi:wsgi app_logs/

That ensures that all the files created in this folder will have wsgi as owner and will be writable for the group that the file belongs. I had to do that because I noticed that the log file created by django app had root as owner and owner group but the application runs through wsgi user.

Finally I changed DEBUG_LOG_DIR to /var/log/app_logs/django_debug.log

2
  • 11
    Keep in mind that if you ever need to rebuild your instance this will need to be performed again manually. This will also not be applied to autoscaled instances. May 31, 2018 at 12:45
  • You can just write the commands in your .ebextensions as a command to execute before application starts to make sure it is always on
    – Josh
    Sep 11, 2021 at 8:45
9

There's a simple way that doesn't require any beanstalk configuration.

In your django settings under LOGGING set up a handler directed to the file '/opt/python/log/{log_file_name}'. The logs can then be accessed via the beanstalk environment menu under "Logs".

LOGGING = {
    ...,
    'handlers': {
        'logfile': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/opt/python/log/{log_file_name}',
        },
    },
    'loggers': {
        'debugger': {
            'level': 'DEBUG',
            'handlers': ['logfile'],
        'propagate': False,
    },
}

This location is stated in the documentation here:

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-instancelocation

5
  • 2
    I agree that this is the ideal solution. But When I try to implement this I'm getting an error stating that permission is denied to the generated log file (django.log). If I SSH into the box and chmod 777 the django.log file it works fine. However that is not an acceptable solution. Nov 28, 2018 at 4:09
  • If you're running into a permission issue like that, one way is to make an EC2 launch script with write permissions
    – bewestphal
    May 31, 2019 at 5:41
  • 1
    Which makes the first sentence of the answer, "There's a simple way that doesn't require any beanstalk configuration", incorrect. Sep 11, 2019 at 21:04
  • @JamesParker: possible cause for your permission error below.
    – djvg
    Mar 5, 2020 at 15:59
  • Does AL2 have a similar location?
    – Nick Brady
    Sep 3, 2020 at 22:28
7

-- edit --

This answer was originally written for Amazon Linux AMI, which has now reached end-of-life.

To keeps things clear and separate, I wrote a new answer for Amazon Linux 2.

-- original answer --

Summary

The simplest solution, in my opinion, is to log to the /opt/python/log folder, as suggested by bewestphal and by @thierry-j (under steve-dunlop's answer).

That is also what the official AWS EB Python Sample Application does: see python-v1.zip source

The log file will then be included automatically when you request logs from EB.

This solution works out-of-the-box, without any modification of .ebextensions, as long as you do not call django-admin.py (or other django code) in your .ebextensions.

However, most apps do need to call django-admin.py in .ebextensions, e.g. in order to migrate. That will cause the log file to be created prematurely, with root owner and root group. This leads to permission errors, because the app runs as wsgi:wsgi.

This can be fixed by adding a new command, at the end of your container_commands, to remove the "premature" log file, e.g.:

container_commands:
  ...
  9999_remove_root_log_file:
    command: rm /opt/python/log/django.log
    ignoreErrors: true

Details below.

Background

On a standard pre-configured Amazon Linux/Python platform, which uses Apache with mod_wsgi (see AWS platform docs), the WSGIDaemonProcess for the Django app runs as user wsgi and group wsgi (see /etc/httpd/conf.d/wsgi.conf on your EC2 instance).

In addition, the default folder permissions for the /opt/python/log folder (on my standard EC2 instance) are: drwxrwxr-x 3 root wsgi 4096 Mar 5 14:08 .

That is, the wsgi group has all permissions (rwx), so the Django app (group wsgi) can create log files there.

This works, out-of-the-box, as demonstrated by the official AWS EB Python Sample Application (python-v1.zip).

However, if you do anything in your .ebextensions that causes the logging file-handler to be initialized (like calling django-admin.py), it will break.

Permission issues

Here's how using django-admin.py in .ebextensions breaks your log file permissions:

Elastic Beanstalk container_commands, in .ebextensions, are executed as the root user (see aws docs).

If you call django-admin.py in any of the container_commands, e.g. with collectstatic or migrate, that will cause your logging file handler(s) to be initialized. If the specified log file does not exist yet, at that time, it will be created, with root owner and root group.

That means the Django app, running as part of the wsgi group, will not have permission to write to the log file (which belongs to the root group).

This leads to permission errors, e.g.: PermissionError: [Errno 13] Permission denied: '/opt/python/log/django.log'

How to reproduce

The following snippet illustrates the permissions issue and shows how to fix it.

To reproduce the issue, add these container_commands to a clean project (e.g. following the AWS EB Django tutorial), configure Django settings.py to log to /opt/python/log/django.log, deploy to AWS EB, then check the eb-activity.log to see the output of the container commands.

...

container_commands:
  0100_show_current_user:
    # show that we are running as root user
    command: whoami
  0200_try_to_remove_log_file:
    # we need a clean slate for this example (make sure no log file owned by wsgi is present)
    command: rm /opt/python/log/django.log
    ignoreErrors: true
  0300_break_log_file_permissions:
    # this causes a new log file to be created, owned by root:root (instead of wsgi:wsgi)
    command: django-admin.py
  0400_show_log_file_permissions:
    # prove that a log file was created by root, and show folder permissions
    command: ls -la /opt/python/log
  0500_fix_by_removing_log_file_after_all_django_admin_calls:
    # remove the log file created by django-admin.py, to ensure that a new log file will  
    # be created when the server starts, owned by wsgi:wsgi
    command: rm /opt/python/log/django.log
    ignoreErrors: true

DRY solution

So, there is no need to mess with file/folder permissions explicitly.

If you don't call django code in .ebextensions, logging to /opt/python/log works, out-of-the-box.

If you do call django code in .ebextensions, e.g. django-admin.py collectstatic, simply remove the log file at the end of your container_commands section.

NOTE: If you want to log files to persist between deployments, only remove them if they are owned by root.

Here's a DRY example:

In .ebextensions config:

option_settings:
  # create EB environment property for the log file path
  aws:elasticbeanstalk:application:environment:
    LOG_FILE_PATH: /opt/python/log/django.log
...

container_commands:
  ...
  # django code called here, e.g. "django-admin.py collectstatic"
  ...
  9999_remove_any_existing_django_log_files:
    command: rm $LOG_FILE_PATH      
    ignoreErrors: true

and in settings.py:

...
# get log path from environment variable, with fallback for local development
log_file_path = os.getenv('LOG_FILE_PATH', 'local.log')
# use this as 'filename' for the file handler, as described in the other answers
...
5
  • PLEASE NOTE: This works (for me) on Amazon Linux AMI, but I haven't tested it (yet) on Amazon Linux 2
    – djvg
    Jul 31, 2020 at 10:21
  • It does not work on Amazon Linux 2 out of the box. Possibly because there's no /opt/python/log (or /opt/python for that matter). I'm still trying to see where the recommended place for logs are. I will probably fall back on /var/log/, creating an app-logs subfolder.
    – Oded
    Aug 6, 2020 at 1:15
  • @Oded: According to the AWS docs about migrating to Amazon Linux 2, the application now resides in /var/app/current. See the Python section.
    – djvg
    Aug 6, 2020 at 18:22
  • Got it. I actually like not having my logs clobbered when the app is reinstalled, so I can compare what happened in a previous version when debugging a problem.
    – Oded
    Aug 6, 2020 at 20:52
  • 1
    @Oded: I'm not sure if there is an alternative for /opt/python/log that is equally convenient. I could not find any recommendations in the AWS docs, but the official Python sample application for EB on Amazon Linux 2 now logs to the /tmp folder, and uses custom logging tasks to include these logs. See updated answer for details.
    – djvg
    Aug 7, 2020 at 16:55
4

This answer is for Amazon Linux 2 only. For those who have not migrated yet, please see my old answer for Amazon Linux AMI.

Background

The official AWS Python sample-application for Amazon linux 2 uses the /tmp folder for logging.

However, custom log files added to /tmp are not included automatically when requesting logs from Elastic Beanstalk. To include custom log files, we need to create logging tasks in subfolders of /opt/elasticbeanstalk/tasks on the EC2 instance. See instructions in the documentation.

The sample app (source) accomplishes this using .ebextensions. However, the AWS Linux 2 migration docs suggest we should use .platform hooks instead:

We recommend using platform hooks to run custom code on your environment instances. You can still use commands and container commands in .ebextensions configuration files, but they aren't as easy to work with. For example, writing command scripts inside a YAML file can be cumbersome and difficult to test.

This has the additional advantage that output from platform hooks is collected in a separate log file, viz. /var/log/eb-hooks.log, which makes debugging a bit easier.

DRY logging setup for a basic Django app on Amazon Linux 2

Log level and log path are defined in one place, as Elastic Beanstalk environment properties, e.g. in .ebextensions/options.config:

option_settings:
  aws:elasticbeanstalk:application:environment:
    LOG_LEVEL: INFO
    DJANGO_LOG_FILE_PATH: /tmp/django-app.log
    ...

The DJANGO_LOG_FILE_PATH environment property can now be used in a platform hook to create logging tasks:

.platform/hooks/postdeploy/020_create_logging_tasks.sh

#!/bin/bash
TASKS_DIR=/opt/elasticbeanstalk/tasks
# include all app log files in bundle logs (replaces ".log" by "*")
echo "${DJANGO_LOG_FILE_PATH//.log/*}" > "$TASKS_DIR/bundlelogs.d/01-app-log.conf"
# include current app log file in tail logs
echo $DJANGO_LOG_FILE_PATH > "$TASKS_DIR/taillogs.d/01-app-log.conf"

Note that the platform hooks require execution permission, e.g. chmod +x 020_create_logging_tasks.sh. On windows you can use git as described here.

To prevent permission issues, we use another platform hook to ensure that the log file is always owned by webapp. Note this hook runs before the logging-tasks hook:

.platform/hooks/postdeploy/010_create_log_file.sh

#!/bin/bash

if test -f "$DJANGO_LOG_FILE_PATH";
then
  echo "$DJANGO_LOG_FILE_PATH exists"
else
  # create log file
  touch $DJANGO_LOG_FILE_PATH
fi

# set log file owner (we are currently "root", but the app runs as "webapp")
chown webapp:webapp $DJANGO_LOG_FILE_PATH

We also use the LOG_LEVEL and DJANGO_LOG_FILE_PATH environment properties in our Django settings:

settings.py

...
# basic logging with file rotation ()
log_level = os.getenv('LOG_LEVEL', 'INFO')
handlers = dict(file={'class': 'logging.handlers.TimedRotatingFileHandler',
                      'filename': os.getenv('DJANGO_LOG_FILE_PATH'),
                      'when': 'midnight',
                      'interval': 1,
                      'backupCount': 1,
                      'encoding': 'utf-8'})
loggers = dict(django=dict(level=log_level, handlers=['file']),
               myapp=dict(level=log_level, handlers=['file']))
LOGGING = dict(version=1,
               disable_existing_loggers=False,
               handlers=handlers,
               loggers=loggers)
...

Some notes:

  • We normally specify custom formatters as well, but I left those out for clarity.

  • The application itself can now be found on the EC2 instance in /var/app/current. Also see extending EB Linux platforms for more details.

  • The application now runs as webapp with group webapp.

  • eb ssh is your friend. See docs.

  • We also use platform hooks to run Django's migrate and collectstatic commands, as described here.

EDIT:

As pointed out by @hax0 in the comments, file permission issues may arise if you try to run manage.py commands on the EC2 instance, using SSH, after deployment.

For example, when using eb ssh, you are logged in as ec2-user, but the log file is owned by the webapp user, and, by default, only the owner has write permission (644). Thus, when running python manage.py as ec2-user, you will get an error saying it cannot configure the log file handler because permission is denied.

A quick & dirty workaround is to change file permissions temporarily, e.g. using

sudo chmod 646 /tmp/django-app.log

Another workaround would be to run manage.py as the webapp user, for example like so:

sudo su - webapp <<'EOF'
source $(find /var/app/venv/*/bin/activate)
export $(/opt/elasticbeanstalk/bin/get-config --output YAML environment | 
         sed -r 's/: /=/' | xargs)
python3 /var/app/current/manage.py showmigrations
EOF

8
  • Deployment works but when we ssh into the instance and try to python manage.py shell, it gives permission error on /tmp/django-app.log file
    – Hax0
    Aug 3, 2021 at 6:32
  • @hax0: Yes, I can confirm this. It happens because you are ec2-user, whereas the log file belongs to webapp (user and group). By default, only the log file owner (i.e. webapp) has write permission (644). If this is incidental, a quick & dirty workaround is to change file permissions temporarily, e.g. sudo chmod 646 /tmp/django-app.log. If it is a structural thing, you could do something similar in a .platform hook. Do keep in mind any security implications. A better alternative might be to try to run the command as webapp.
    – djvg
    Aug 3, 2021 at 9:03
  • @hax0: manage.py shell works for me after changing file permissions. I did manage to run other manage.py commands as the webapp user, but could not get manage.py shell to work that way. See edit in answer.
    – djvg
    Aug 3, 2021 at 11:29
  • Using this exact solution on Amazon Linux 2, deployment fails (Gunicorn + Django + Nginx) with a Permission denied: '/tmp/django-app.log'. Perhaps that is because of the manage.py commands being run in container_commands. Dec 28, 2021 at 22:50
  • django-app.log appears to be mode 0600, owned by root:root. I believe that's because some other command must be run by the root user, which generates that file as owned by root. (Again, assuming that's the stuff in container_commands but maybe something else.) Dec 28, 2021 at 22:51
1

As a beginner in terms of linux permissions, it took me some time to get it to work. Summarising the above given answers the following finally worked for me:

logging.config

commands:
  00_create_dir:
    command: mkdir -p /var/log/app-logs
  01_change_permissions:
    command: chmod g+s /var/log/app-logs
  02_change_default_owner:
    command: setfacl -d -m g::rw /var/log/app-logs
  03_change_owner:
    command: chown wsgi:wsgi /var/log/app-logs

settings.py

LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
    'file': {
        'level': 'DEBUG',
        'class': 'logging.FileHandler',
        'filename': '/var/log/app-logs/django.log',
    },
},
'loggers': {
    'django': {
        'handlers': ['file'],
        'level': 'DEBUG',
        'propagate': True,
    },
},
}

With this I can see the logs as a separate section using 'eb logs' or within the Beanstalk environment, section "logs".

0

If you're using gunicorn, then you can use this flag:

gunicorn --access-logfile=- 
0

Here is my solution on 64bit Amazon Linux 2023/4.0.2

commands:
  00_create_dir:
    command: mkdir -p /var/log/app-logs
  01_change_permissions:
    command: chmod g+s /var/log/app-logs
  02_change_owner:
    command: chown webapp:webapp /var/log/app-logs

and the log file setting in Django,

'handlers': {
    'console': {
        'level': 'INFO',
        'class': 'logging.StreamHandler',
    },
    'logfile': {
        'level': 'INFO',
        'class': 'logging.handlers.RotatingFileHandler',
        'filename': "/var/log/app-logs/django.log",
        'formatter': 'standard',
    },
},
'loggers': {
    '': {
        'handlers': ['console', 'logfile'],
        'level': 'INFO',
    },
}

Make sure that the log file has the correct ownership and group configuration, the owner and group should be webapp:wehapp as follows

[ec2-user@ip-171-11-1-8 app-logs]$ ll django.log
-rw-r--r--. 1 webapp webapp 1120 Jul 29 01:08 django.log
-4

By default in elasticbeanstalk, you can see the django error logs here.

/var/log/httpd/error_log
3
  • No, Django server errors do not show up in this file.
    – Thierry J.
    Oct 26, 2017 at 12:42
  • they might end up there, but that's only on some cases. Nov 30, 2017 at 2:30
  • See this discussion on AWS forums.
    – djvg
    Mar 4, 2020 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.