89

Inside my docker-compose.yml, I have the following service healthcheck section. I want to know if MariaDB is actually ready to handle queries. A service named cmd is configured to depend on condition: service_healthy.

  db:
    image: mariadb:10
    environment:
      MYSQL_RANDOM_ROOT_PASSWORD: 1
      MYSQL_USER: user
      MYSQL_PASSWORD: password
      MYSQL_DATABASE: database
    healthcheck:
      test: ["CMD", "mysql", "--user=user", "--password=password", "--execute='SELECT 1'", "--host=127.0.0.1", "--port=3306"]
      interval: 1s
      retries: 30

This healthcheck does not work, shows that the service is unhealthy.

How do I check the output of the test CMD?

3 Answers 3

187

You can use:

docker inspect --format "{{json .State.Health }}" <container name> | jq

Output:

{
    "Status": "unhealthy",
    "FailingStreak": 63,
    "Log": [
        {
            "Start": "2017-03-11T20:49:19.668895201+03:30",
            "End": "2017-03-11T20:49:19.735722044+03:30",
            "ExitCode": 1,
            "Output": "ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''SELECT 1'' at line 1\n"
        }
    ]
}

And look for the output section.

To get the Output only:

docker inspect --format "{{json .State.Health }}" mariadb_db_1 | jq '.Log[].Output'

For swarm mode use the folling format (thanks for pointing it out @shotgunner):

{{json.Spec.TaskTemplate.ContainerSpec.Healthcheck}}

Feel free to swap jq for whatever tool you use for json pretty print.

4
  • 1
    Hi, do you know any way to attach this logs to the same json-logfile that writes entrypoint? I'd like to see healthcheck logs executing docker logs, all of them in the same place Jan 11, 2019 at 10:10
  • 2
    In swarm mode and in manager {{json .State.Health }} not works. Use {{json .Spec.TaskTemplate.ContainerSpec.Healthcheck }} instead Mar 2, 2020 at 7:01
  • 2
    If using jq anyway, all querying can be done through it; it's shorter and nicer: docker inspect $CONTAINER | jq '.[].State.Health'.
    – Alex Povel
    Mar 31, 2021 at 16:10
  • I wonder if the format of the first command is correct. It boasts a suspicious blank space.
    – Stephane
    Jun 9, 2021 at 8:06
6

docker-compose ps will indicate the status of each service, including its health if healthcheck is defined. It's good for a basic overview.

% docker-compose ps
                Name                                Command                       State                                       Ports                            
----------------------------------------------------------------------------------------------------------------------------------------------------------------
remix-theme-editor_analytics_1            /bin/sh -c /analytics/run. ...   Up                                                                                   
remix-theme-editor_base_1                 /bin/bash                        Exit 0                                                                               
remix-theme-editor_flower_1               /entrypoint --environment  ...   Exit 137                                                                             
remix-theme-editor_frontend_1             /bin/sh -c perl -p -i -e ' ...   Exit 137                                                                             
remix-theme-editor_js-app_1               npm run                          Exit 0                                                                               
remix-theme-editor_mq_1                   docker-entrypoint.sh rabbi ...   Up (healthy)            15671/tcp, 15672/tcp, 25672/tcp, 4369/tcp, 5671/tcp, 5672/tcp
remix-theme-editor_mysql-migration_1      /entrypoint_mysql-migratio ...   Exit 0                                                                               
remix-theme-editor_mysql_1                /bin/sh -c /entrypoint_wra ...   Up (health: starting)   127.0.0.2:3308->3306/tcp                                     
remix-theme-editor_page-renderer_1        npm run start:watch              Up                                                                                   
remix-theme-editor_python-app_1           /entrypoint                      Exit 2                                                                               
remix-theme-editor_redis_1                docker-entrypoint.sh /bin/ ...   Up (health: starting)   6379/tcp                                                     
remix-theme-editor_scheduler_1            /entrypoint --environment  ...   Exit 137                                                                             
remix-theme-editor_socket_1               /entrypoint --environment  ...   Exit 1                                                                               
remix-theme-editor_static-builder_1       npm run watch                    Up                                                                                   
remix-theme-editor_static-http_1          nginx -g daemon off;             Up                      127.0.0.2:6544->443/tcp, 80/tcp                              
remix-theme-editor_web_1                  /entrypoint --environment  ...   Exit 1                                                                               
remix-theme-editor_worker_1               /entrypoint --environment  ...   Exit 1                                                                               
remix-theme-editor_worker_screenshots_1   /entrypoint --environment  ...   Exit 1     

If you want more details, use docker inspect in conjuction with docker ps -q <service-name>.

% docker inspect --format "{{json .State.Health }}" $(docker-compose ps -q mq) | jq
{
  "Status": "starting",
  "FailingStreak": 48,
  "Log": [
    {
      "Start": "2018-10-03T00:40:18.671527745-05:00",
      "End": "2018-10-03T00:40:18.71729051-05:00",
      "ExitCode": -1,
      "Output": "OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused \"exec: \\\"nc\\\": executable file not found in $PATH\": unknown"
    },
...

You can always debug the healthcheck yourself by simply executing your healthcheck code yourself. For example:

% docker exec -it $(docker-compose ps -q socket) nc -w2 127.0.0.1 5672
(UNKNOWN) [127.0.0.1] 5672 (?) : Connection refused

You can also do the same in shell:

% docker exec -it $(docker-compose ps -q socket) bash
root@b5da5207d344:~/src# nc -w2 127.0.0.1 5672
(UNKNOWN) [127.0.0.1] 5672 (?) : Connection refused
root@b5da5207d344:~/src# echo $?
1

Finally, you can simply use docker-compose up in the first terminal window, and docker-compose logs -f in another. This will display all logs from docker-compose-managed containers.

1
  • Hi, do you know any way to attach this logs to the same json-logfile that writes entrypoint? I'd like to see healthcheck logs executing docker logs, all of them in the same place Jan 11, 2019 at 10:09
1

In Swarm Mode:

  1. First use docker service ps service_name in the manager to find the failed task id and the corresponding node.
manager$ docker service ps service_name
ID                  NAME                 IMAGE            NODE                DESIRED STATE       CURRENT STATE               ERROR                              PORTS
liwww3qzg9dz        service_name.1       image_name:1.3   s-3                 Running             Running 27 seconds ago                                         
hcgxmwk2efj0         \_ service_name.1   image_name:1.3   s-3                 Shutdown            Failed about a minute ago   "task: non-zero exit (137): do…"  

In this example, hcgxmwk2efj0 is the task id and s-3 is the node name.

  1. Then use docker inspect --format "{{json .Status.ContainerStatus.ContainerID }}" task_id in the manager to get the container id.
manager$ docker inspect --format "{{json .Status.ContainerStatus.ContainerID }}" hcgxmwk2efj0
"412b09d5244047b31471248fd9a0807e5ea42406fb8f5b1701df2244933e30c8"
  1. Then ssh to that node, and use the command docker inspect --format "{{json .State.Health }}" container_id | jq to get the log of the healthcheck. (| jq is not necessary in this command)
s-3$ docker inspect --format "{{json .State.Health }}" 412b09d5244047b31471248fd9a0807e5ea42406fb8f5b1701df2244933e30c8 | jq
{
  "Status": "unhealthy",
  "FailingStreak": 3,
  "Log": [
    {
      "Start": "2021-09-07T06:10:05.233163051Z",
      "End": "2021-09-07T06:10:07.585487343Z",
      "ExitCode": 0,
      "Output": "... log 1 ..."
    },
    {
      "Start": "2021-09-07T06:10:37.644936244Z",
      "End": "2021-09-07T06:10:39.881196276Z",
      "ExitCode": 0,
      "Output": "... log 2 ..."
    },
    {
      "Start": "2021-09-07T06:11:10.16172012Z",
      "End": "2021-09-07T06:11:25.161912411Z",
      "ExitCode": -1,
      "Output": "Health check exceeded timeout (15s)"
    },
    {
      "Start": "2021-09-07T06:11:55.297395088Z",
      "End": "2021-09-07T06:12:10.302928565Z",
      "ExitCode": -1,
      "Output": "Health check exceeded timeout (15s)"
    },
    {
      "Start": "2021-09-07T06:12:40.371234778Z",
      "End": "2021-09-07T06:12:55.371393914Z",
      "ExitCode": -1,
      "Output": "Health check exceeded timeout (15s)"
    }
  ]
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.