168

I have a Docker container running in a host of 1G RAM (there are also other containers running in the same host). The application in this Docker container will decode some images, which may consume memory a lot.

From time to time, this container will exit. I doubt it is due to out of memory but not very sure. I need a method to find the root cause. So is there any way to know what happened for this container's death?

12
  • 13
    You can check the logs for that container via docker logs <container-id>.
    – techtabu
    Jun 30, 2016 at 2:30
  • 6
    but the container has exited, i guess I can not logs it any more ?
    – Li Bin
    Jun 30, 2016 at 2:32
  • Just tried on my machine. You can still access the logs even when the container has exited.
    – Samuel Toh
    Jun 30, 2016 at 2:36
  • Did you at least try?
    – techtabu
    Jun 30, 2016 at 2:36
  • techtabu, yes I did . It doesn't help anyway
    – Li Bin
    Jun 30, 2016 at 2:37

5 Answers 5

196

Others have mentioned docker logs $container_id to view the output of the application. This would always be my first thing to check.

Next, you can run a docker inspect $container_id to view details on the state, e.g.:

    "State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 2,
        "Error": "",
        "StartedAt": "2016-06-28T21:26:53.477229071Z",
        "FinishedAt": "2016-06-28T21:26:53.478066987Z"
    },

The important line there is "OOMKilled" which will be true if you exceed the container memory limits and Docker kills your app. You may also want to lookup the exit code to see if it identifies a cause for the exit by your app.

Note, this only indicates if docker itself kills your process, and requires that you have set a memory limit on your container. Outside of docker, the Linux kernel can kill your process if the host itself runs out of memory. Linux often writes to a log in /var/log when this happens. With Docker Desktop on Windows and Mac, you can adjust the memory allocated to the embedded Linux VM in the docker settings.

6
  • 18
    I don't understand here is since my container is gone, how the "inspect" will work ? From discussion above, once the app dies , the container will dies too. You mean restart the same image then inspect ?
    – Li Bin
    Jun 30, 2016 at 2:59
  • 14
    @LiBin a container don't get wipe away when it dies, it simply gets to a halt state like status= stopped or exited. 'docker ps -a' and see for yourself
    – Samuel Toh
    Jun 30, 2016 at 3:04
  • I was getting exit 0 every time running a memory intensive operation and OOMKilled was false. Increasing memory made it working again.
    – Andy
    Apr 19, 2018 at 13:47
  • 1
    This can happen if the Linux kernel, rather than the docker engine, kills processes in the container. You will often see that in the OS logs under /var/log on the host.
    – BMitch
    Feb 26, 2019 at 16:48
  • 2
    I know it's a typo, but having the linux kernel "lol" your process seems so appropriate
    – Jim T
    Mar 5, 2022 at 11:14
12

You can find out whether the process inside the container was OOMkilled by reading the logs. OOMkills are initiated by the kernel so every time it happens there's a bunch of lines in /var/log/kern.log, for example:

python invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
oom_kill_process+0x22e/0x450
Memory cgroup out of memory: Kill process 31204 (python) score 1994 or sacrifice child
Killed process 31204 (python) total-vm:7350860kB, anon-rss:4182920kB, file-rss:2356kB, shmem-rss:0kB
1
  • 1
    This answer helped me find what's wrong with a container that docker would restart on exit (docker inspect does not help much here).
    – m90
    Jul 9, 2020 at 13:34
11

While the accepted answer is the best option, sometimes it can be useful to inspect from the host the content of the journal too (on linux).

You can do that by typing:

sudo journalctl -u docker

or tailing it

sudo journalctl -u docker -f

or piping the output to less if it is too long for your terminal buffer

sudo journalctl -xn -u docker | less
1
  • 1
    That was it! Great tip. No logs anywhere from Docker, but this showed me that something killed and restarted the Daemon process.
    – oligofren
    Aug 14, 2021 at 8:53
0

docker inspect -f '{{ .State }}' $container_id and check 8th property which is ExitCode.

0

If you cannot access the container's logs, especially when you use the docker API - e.g. because docker.errors.NotFound: 404 Client Error - the reason could be that the container has been automatically removed after exiting.

It can be caused by running the container with docker run --rm option, which is: Automatically remove the container when it exits. You will be not able to get the logs any more then.

Just get rid of --rm option and take care for removing the container manually after collecting the logs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.