24

I have a project using the official nginx docker container from Docker Hub, launching via Docker Compose. I have healthchecks configured in Docker Compose for each of my containers, and recently the healthcheck for this nginx container has been behaving strangely; on launching with docker-compose up -d, all my containers launch, and begin running healthchecks, but the nginx container looks like it never runs the healthcheck. I can manually run the script just fine if I docker exec into the container, and the healthcheck runs normally if I restart the container.

Example output from docker ps:

CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS                            PORTS                                                                       NAMES
458a55ae8971        my_custom_image           "/tini -- /usr/local…"   7 minutes ago       Up 7 minutes (healthy)                                                                                        project_worker_1
5024781b1a73        redis:3.2                 "docker-entrypoint.s…"   7 minutes ago       Up 7 minutes (healthy)            127.0.0.1:6379->6379/tcp                                                    project_redis_1
bd405dde8ce7        postgres:9.6              "docker-entrypoint.s…"   7 minutes ago       Up 7 minutes (healthy)            127.0.0.1:15432->5432/tcp                                                   project_postgres_1
93e15c18d879        nginx:mainline            "nginx -g 'daemon of…"   7 minutes ago       Up 7 minutes (health: starting)   127.0.0.1:80->80/tcp, 127.0.0.1:443->443/tcp                                nginx

Example (partial, for brevity) output from docker inspect nginx:

    "State": {
        "Status": "running",
        "Running": true,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 11568,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2018-02-13T21:04:22.904241169Z",
        "FinishedAt": "0001-01-01T00:00:00Z",
        "Health": {
            "Status": "unhealthy",
            "FailingStreak": 0,
            "Log": []
        }
    },

The portion of the docker-compose.yml defining the nginx container:

nginx:
  image: nginx:mainline
  # using container_name means there will only ever be one nginx container!
  container_name: nginx
  restart: always
  networks:
    - proxynet
  volumes:
    - /etc/nginx/conf.d
    - /etc/nginx/vhost.d
    - /usr/share/nginx/html
    - tlsdata:/etc/nginx/certs:ro
    - attachdata:/usr/share/nginx/html/uploads:ro
    - staticdata:/usr/share/nginx/html/static:ro
    - ./nginx/healthcheck.sh:/bin/healthcheck.sh
  healthcheck:
    test: ['CMD', '/bin/healthcheck.sh']
    interval: 1m
    timeout: 5s
    retries: 3
  ports:
    # Make the http/https ports available on the Docker host IPv4 loopback interface
    - '127.0.0.1:80:80'
    - '127.0.0.1:443:443'

The healthcheck.sh I am loading in as a volume:

#!/bin/bash

service nginx status || exit 1

It looks like the problem is just an issue with systemd never returning from the status check when the container initially launches, and at the same time the configured healthcheck timeout does not trigger. Everything else works, and nginx is up and responding, but it would be nice for the healthcheck to function properly without needing to manually restart each time I start up.

Is there something missing in my configuration, or a better check I can run?

2
  • Have you checked less /var/log/nginx/error.log? What is showing after first service nginx status? Commented May 16, 2018 at 14:08
  • The nginx official image symlinks error.log to /dev/stderr, which is captured by Docker. So running less /var/log/nginx/error.log will not do what you expect. Regardless, yes, I have checked the log outputs, and there are no errors. As the question states, nginx launches and functions normally. The issue is the healthcheck script never runs; i.e. there is no first service nginx status run.
    – wmorrell
    Commented May 17, 2018 at 17:05

4 Answers 4

26

I think that there is no need for a custom script in this case.

Try just change your healthcheck test to

test: ["CMD", "service", "nginx", "status"]

That works fine for me.

Try to use " instead of ' as well, just in case :)

EDIT

If you really want to force an exit 1, in case of failure, you could use:

test: service nginx status || exit 1
3
  • The custom script is necessary because the healthcheck mechanism expects a 0 exit status for healthy and a 1 exit status for unhealthy. service does not match these expectations, hence the script that forces a return of 1 for unhealthy.
    – wmorrell
    Commented Jan 16, 2019 at 19:15
  • Again, this is not a solution. The problem as far as I can tell is that the healthcheck for this container never runs on first launch. It will only run if the container is manually restarted. It runs fine after restart. Also, using a string like your edit runs the healthcheck in a subshell, e.g. sh -c 'service nginx status || exit 1' which is not the right thing.
    – wmorrell
    Commented Jan 17, 2019 at 22:22
  • 12
    BTW, this does not work for the alpine version as service isn't a command.
    – gunr2171
    Commented Mar 16, 2019 at 0:39
13

for the official alpine nginx image you can also do:

healthcheck:
      test: ["CMD-SHELL", "wget -O /dev/null http://localhost || exit 1"]
      timeout: 10s

wget is part of the standard image. What this does is download your index.html/php/whatever to nowhere (/dev/null), and it should timeout and fail otherwise.

5
  • This is not relevant to the question. The problem was with healthchecks never running in the first place, not alternate ways to do a healthcheck.
    – wmorrell
    Commented Jun 3, 2020 at 22:17
  • 2
    wget is NOT part of official nginx image. Commented Jun 13, 2020 at 8:34
  • 3
    @TheGodfather I am currently using the nginx:1.19-alpine minimal image and it contains wget as well as curl. I implemented my test using test: ["CMD-SHELL", "curl -so /dev/null http://localhost/ || exit 1"] and it is working fine, however wget shows me (connection refused). Commented Feb 27, 2021 at 14:18
  • 2
    Small clarification: official nginx:latest images do NOT include wget while nginx:alpine includes it (that's weird, typically one would expect alpine to be somehow "subset" of full image) Commented Feb 28, 2021 at 8:07
  • 3
    This is not surprising once you know that Alpine is based on Busybox, which is a single binary that is the tool under whose name it is called. Sounds strange? Try ls -l /usr/bin/ on Alpine. ;)
    – wedi
    Commented Jan 23, 2022 at 19:50
5

Over a year later, I have found a solution. First, an additional clarification on the environment, what I believe is happening, and speculation on a possible bug with the Docker Engine.

The Compose file I am using now is launching a lightly modified version of the 'official' Alpine NGINX image, which uses COPY to load in the healthcheck script and adds HEALTHCHECK explicitly in the image. This image is used for an nginx service, and is used in concert with an image running jwilder/docker-gen to use container metadata from Docker to generate NGINX configuration files. This container is running as a service named nginx-gen. When containers change, configuration is re-generated, and if there are any changes, a SIGHUP is sent to the nginx service.

What I discovered is the following:

  • If all services are launched together, the nginx service never runs healthchecks;
  • If the nginx service is restarted soon after launch, healthchecks complete normally;
  • If the nginx service is launched by itself, healthchecks complete normally;
  • If all services other than nginx-gen are launched together, healthchecks complete normally;
  • If all services are launched together, but nginx-gen is modified to sleep 60 before doing anything, healthchecks complete normally;

So, it appears that there is some obscure interaction with signal processing, Docker, and NGINX. If a SIGHUP is sent to an NGINX process in a container before the first healthcheck runs in that container, no healthchecks ever run.

The final iteration I came up with modifies the nginx-gen container to poll the health of the nginx container. It looks up the health status of a container with a defined label in a loop, with a short sleep. Once the nginx container reports healthy, nginx-gen proceeds to generate configuration files. I also changed the notification method to docker exec a script to explicitly test and reload configuration in the nginx container, rather than rely on SIGHUP.

End result: I can docker-compose up -d, and everything eventually reports healthy without further intervention. Success!

1
  • 2
    So for even more tl;dr, how did your nginx service in docker-compose look like in the end? And what was changed comparing to initial code samples?
    – Eugene
    Commented Mar 14 at 9:24
4

I attempted the same script and encountered the same issue. I changed the healthcheck.sh to instead run like this:

#!/bin/bash

if service nginx status; then
    exit 0
else
    exit 1
fi

Running this in the docker container resulted in successful health checks.

1
  • Thanks, glad it's working for you; but, I do not think this is the correct solution. When I try this script, I still see the same behavior. The container is perpetually in starting state, and inspecting the container shows a health status of unhealthy with a FailingStreak value of 0. The healthcheck script simply never runs, regardless of content.
    – wmorrell
    Commented Jun 12, 2018 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.