12

Docker exited for no apparent reason and I'm trying to understand what happened.

Right now the docker daemon is a loaded service, inactive with an exit status 0 SUCCESS.

$ systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Tue 2020-12-01 06:25:16 UTC;
     Docs: https://docs.docker.com
 Main PID: 2218 (code=exited, status=0/SUCCESS)

Looking at the docker logs, this happened because it processed signal 'terminated':

$ journalctl -u docker.service | tail -25
Nov 30 18:30:21 ip-10-38-4-210 dockerd[2218]: time="2020-11-30T18:30:21.728694550Z" <redacted irrelevant>
Dec 01 06:25:05 ip-10-38-4-210 systemd[1]: Stopping Docker Application Container Engine...
Dec 01 06:25:05 ip-10-38-4-210 dockerd[2218]: time="2020-12-01T06:25:05.867748396Z" level=info msg="Processing signal 'terminated'"
Dec 01 06:25:16 ip-10-38-4-210 systemd[1]: Stopped Docker Application Container Engine.

No user was logged in on that host at that time, no user explicitly terminated the docker daemon.

  • Why is this happening, what logs could provide a clue?
  • Can I configure dockerd to restart automatically upon exits?

This is Docker version 18.09.7 (build 2d0083d) on Ubuntu 16.04.6 LTS on x86-64.

The relationship with systemd is being asked on Unix.stackexchange here

4

1 Answer 1

5
Dec 01 06:25:05 ip-10-38-4-210 systemd[1]: Stopping Docker Application Container Engine...

At this point, systemd sent docker a stop request.

Dec 01 06:25:05 ip-10-38-4-210 dockerd[2218]: time="2020-12-01T06:25:05.867748396Z" level=info msg="Processing signal 'terminated'"

Docker received that stop request and exits gracefully.

Dec 01 06:25:16 ip-10-38-4-210 systemd[1]: Stopped Docker Application Container Engine.

Systemd now reports the stop has finished. The 10 seconds is likely from containers that did not gracefully handle the docker container stop command and were killed after 10 seconds, then another second to finish processing the request.

So from the logs provided, the dockerd daemon is functioning as requested and you should check what would have sent a systemctl stop docker command.

Can I configure dockerd to restart automatically upon exits?

dockerd doesn't control systemd, systemd controls dockerd, so you'd need to solve the issue from the systemd side.


In this specific case, the thing telling systemd to stop the docker service appears to be the Ubuntu unattended-update service applying updates the the Ubuntu version of the containerd package.

https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1870514

My recommended fix would be to install docker from the upstream Docker repositories which does not appear to have this issue:

https://docs.docker.com/engine/install/ubuntu/

4
  • How do I dig into why systemd sent docker a terminate signal?
    – mipnw
    Dec 1, 2020 at 13:36
  • 1
    @mipnw No idea if systemd gives logs on that, might be worth looking around /var/log for that timestamp. I'm more the docker guy (since the title and tag indicated a docker issue). For systemd help, superuser.SE or unix.SE may be better.
    – BMitch
    Dec 1, 2020 at 14:11
  • ok. I've cross posted on unix.SE and added a link in this question.
    – mipnw
    Dec 1, 2020 at 14:22
  • 1
    Thanks for pointing out that the reason could be the ubuntu patch since i encounter the same issues in all my ubuntu EC2 and the docker daemon stoped at the same time as PO, Dec 01 06:27:22
    – nngeek
    Jan 18, 2021 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.