326

I have a Python (2.7) app which is started in my dockerfile:

CMD ["python","main.py"]

main.py prints some strings when it is started and goes into a loop afterwards:

print "App started"
while True:
    time.sleep(1)

As long as I start the container with the -it flag, everything works as expected:

$ docker run --name=myapp -it myappimage
> App started

And I can see the same output via logs later:

$ docker logs myapp
> App started

If I try to run the same container with the -d flag, the container seems to start normally, but I can't see any output:

$ docker run --name=myapp -d myappimage
> b82db1120fee5f92c80000f30f6bdc84e068bafa32738ab7adb47e641b19b4d1
$ docker logs myapp
$ (empty)

But the container still seems to run;

$ docker ps
Container Status ...
myapp     up 4 minutes ... 

Attach does not display anything either:

$ docker attach --sig-proxy=false myapp
(working, no output)

Any ideas whats going wrong? Does "print" behave differently when ran in background?

Docker version:

Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.2
Git commit (client): a8a31ef
OS/Arch (client): linux/arm
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.2
Git commit (server): a8a31ef

14 Answers 14

577

Finally I found a solution to see Python output when running daemonized in Docker, thanks to @ahmetalpbalkan over at GitHub. Answering it here myself for further reference :

Using unbuffered output with

CMD ["python","-u","main.py"]

instead of

CMD ["python","main.py"]

solves the problem; you can see the output now (both, stderr and stdout) via

docker logs myapp

why -u ref

- print is indeed buffered and docker logs will eventually give you that output, just after enough of it will have piled up
- executing the same script with python -u gives instant output as said above
- import logging + logging.warning("text") gives the expected result even without -u

what it means by python -u ref. > python --help | grep -- -u

-u     : force the stdout and stderr streams to be unbuffered;
8
  • 8
    -u seems to work for me, but is there some documentation somewhere with a description of what it actually does? Dec 6, 2016 at 10:56
  • 17
    As suggested by other answers, you can try setting environment variable ENV PYTHONUNBUFFERED=0 in case the -u flag does not work.
    – Farshid T
    Jan 12, 2017 at 11:32
  • 2
    This was my problem too. For a more detailed explanation, see stackoverflow.com/a/24183941/562883 Apr 15, 2017 at 21:27
  • 3
    Works like a dream on python3, while setting PYTHONUNBUFFERED=0 wasnt helping. Dec 21, 2018 at 20:20
  • 2
    Thanks This helped us to start with debugging on prod
    – Yash Gupta
    Jun 10, 2022 at 7:59
156

In my case, running Python with -u didn't change anything. What did the trick, however, was to set PYTHONUNBUFFERED=1 as environment variable:

docker run --name=myapp -e PYTHONUNBUFFERED=1 -d myappimage

[Edit]: Updated PYTHONUNBUFFERED=0 to PYTHONUNBUFFERED=1 after Lars's comment. This doesn't change the behavior and adds clarity.

7
  • 7
    In my case, adding -e PYTHONUNBUFFERED=0 helps.
    – David Ng
    Dec 8, 2015 at 9:50
  • 1
    Thank you! I was banging my head off a wall for hours, and couldn't get logs to work even with -u. Your solution fixed it for me on Docker for Mac with Django
    – Someguy123
    Sep 21, 2016 at 19:34
  • 2
    i think this is a better solution, that we don't have to rebuild the docker image to see the outputs
    – FF0605
    Oct 29, 2018 at 22:28
  • 5
    This is great thanks. Its worth mentioning that this just needs to be a non empty character to work according to the docs PYTHONUNBUFFERED
    – A Star
    Nov 19, 2018 at 21:23
  • 19
    PYTHONUNBUFFERED=0 is misleading b/c it suggests that unbuffering is disabled. Instead it's enabled b/c python looks for a non-empty string. That said, better use PYTHONUNBUFFERED=1 which has the same effect but doesn't lead to wrong assumptions. Mar 3, 2021 at 16:21
44

If you want to add your print output to your Flask output when running docker-compose up, add the following to your docker compose file.

web:
  environment:
    - PYTHONUNBUFFERED=1

https://docs.docker.com/compose/environment-variables/

3
  • Someone able to elaborate why this variable not works when divined in the Dockerfile and not in the docker-compose file? I thought this shouldn't matter but works like a charm! Nov 23, 2020 at 22:10
  • @0x78f1935 are you perhaps using an ARG directive instead of an ENV one? It should work in the dockerfile as well. Feb 25, 2022 at 16:07
  • 1
    Doesn't work for me :/ Any Idea as to why this might not work? May 9, 2022 at 17:44
39
+75

See this article which explain detail reason for the behavior:

There are typically three modes for buffering:

  • If a file descriptor is unbuffered then no buffering occurs whatsoever, and function calls that read or write data occur immediately (and will block).
  • If a file descriptor is fully-buffered then a fixed-size buffer is used, and read or write calls simply read or write from the buffer. The buffer isn’t flushed until it fills up.
  • If a file descriptor is line-buffered then the buffering waits until it sees a newline character. So data will buffer and buffer until a \n is seen, and then all of the data that buffered is flushed at that point in time. In reality there’s typically a maximum size on the buffer (just as in the fully-buffered case), so the rule is actually more like “buffer until a newline character is seen or 4096 bytes of data are encountered, whichever occurs first”.

And GNU libc (glibc) uses the following rules for buffering:

Stream               Type          Behavior
stdin                input         line-buffered
stdout (TTY)         output        line-buffered
stdout (not a TTY)   output        fully-buffered
stderr               output        unbuffered

So, if use -t, from docker document, it will allocate a pseudo-tty, then stdout becomes line-buffered, thus docker run --name=myapp -it myappimage could see the one-line output.

And, if just use -d, no tty was allocated, then, stdout is fully-buffered, one line App started surely not able to flush the buffer.

Then, use -dt to make stdout line buffered or add -u in python to flush the buffer is the way to fix it.

21

Since I haven't seen this answer yet:

You can also flush stdout after you print to it:

import time

if __name__ == '__main__':
    while True:
        print('cleaner is up', flush=True)
        time.sleep(5)
3
  • 2
    this worked perfectly for me, stupid that this needs to be there, but works great now.
    – james-see
    Dec 23, 2019 at 1:25
  • 3
    This worked for me as well. None of the envar methods, nor the "-u" method worked for me.
    – tlochner95
    Oct 15, 2020 at 19:31
  • 3
    PS: if you have several print and then you flush at the last one, you get to see all the previous print up to the one with flush=True. Jul 15, 2021 at 20:09
20

Try to add these two environment variables to your solution PYTHONUNBUFFERED=1 and PYTHONIOENCODING=UTF-8

2
  • 1
    why would you need PYTHONIOENCODING?
    – MPękalski
    Feb 4, 2021 at 14:50
  • To get away with no ASCII chars. Feb 4, 2021 at 14:59
8

You can see logs on detached image if you change print to logging.

main.py:

import time
import logging
print "App started"
logging.warning("Log app started")
while True:
    time.sleep(1)

Dockerfile:

FROM python:2.7-stretch
ADD . /app
WORKDIR /app
CMD ["python","main.py"]
2
  • 1
    nice. tip: use Python 3.
    – adhg
    Sep 6, 2019 at 15:05
  • question is in Python 2 (print statement without parenthesis) therefore am using 2 here. Although it is exactly the same behaviour on Python3.6 so thanks for a tip ;)
    – The Hog
    Sep 7, 2019 at 7:43
7

If anybody is running the python application with conda you should add --no-capture-output to the command since conda buffers to stdout by default.

ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "my-app", "python", "main.py"]
1
  • Thanks for this. Now it logs to the interactive console as well as when using the docker log command! Sep 5, 2022 at 20:37
5

I had to use PYTHONUNBUFFERED=1 in my docker-compose.yml file to see the output from django runserver.

4

As a quick fix, try this:

from __future__ import print_function
# some code
print("App started", file=sys.stderr)

This works for me when I encounter the same problems. But, to be honest, I don't know why does this error happen.

2
  • Thanks for the tip! Tried replacing all prints with your version, unfortunately it did not work for me, still can't get any output via docker logs (changing between sys.stderr / sys.stdout does has no visible result). Is this a docker bug?
    – jpdus
    Apr 16, 2015 at 11:07
  • See my answer, the reason is: stderr was unbuffered, so you can make it fix with your solution.
    – atline
    Sep 5, 2019 at 9:02
4

If you aren't using docker-compose and just normal docker instead, you can add this to your Dockerfile that is hosting a flask app

ARG FLASK_ENV="production"
ENV FLASK_ENV="${FLASK_ENV}" \
    PYTHONUNBUFFERED="true"

CMD [ "flask", "run" ]
0
4

When using python manage.py runserver for a Django application, adding environment variable PYTHONUNBUFFERED=1 solve my problem. print('helloworld', flush=True) also works for me.

However, python -u doesn't work for me.

1

Usually, we redirect it to a specific file (by mounting a volume from host and writing it to that file).

Adding a tty using -t is also fine. You need to pick it up in docker logs.

Using large log outputs, I did not have any issue with buffer storing all without putting it in dockers log.

0

I use visual studio code with docker extension to create Dockerfile in python automatically.

By default it have this line

# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1

it works and show logs when container run in detached mode, so I think that's the preferred way. I use python 3.11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.