42

We're using Kubernetes 1.1.3 with its default fluentd-elasticsearch logging.

We also use LivenessProbes on our containers to make sure they operate as expected.

Our problem is that lines we send out to the STDOUT from the LivenessProbe does not appear to reach Elastic Search.

Is there a way to make fluentd ship LivenessProbes output like it does to regular containers in a pod?

2 Answers 2

66

The output from the probe is swallowed by the Kubelet component on the node, which is responsible for running the probes (source code, if you're interested). If a probe fails, its output will be recorded as an event associated with the pod, which should be accessible through the API.

The output of successful probes isn't recorded anywhere unless your Kubelet has a log level of at least --v=4, in which case it'll be in the Kubelet's logs.

Feel free to file a feature request in a Github issue if you have ideas of what you'd like to be done with the output :)

3
  • How do you configure the loglevel of a kubelet (for us noobs)? Commented Jan 21, 2020 at 15:11
  • Upgraded my comment to a new question - stackoverflow.com/questions/59844723/… Commented Jan 21, 2020 at 16:24
  • 4
    If you do kubectl describe po <pod name> of the pod with the liveness / readiness probe, you should be able to see the status of the probe in the events messages section Commented Jan 21, 2023 at 0:51
22

Also consider as a workaround. That from the liveness probe (of exec type) you can write to the stdout of the entrypoint to basically write stuff into the output of the kubectl logs <pod name>.

Liveness probe command e.g.

#!/bin/bash

echo "Hello liveness probe stdout"
echo "Hello entrypoint stdout" > /proc/1/fd/1

kubectl logs <pod name>

Hello entrypoint stdout
Hello entrypoint stdout
Hello entrypoint stdout
1
  • 4
    Thanks! That's a great tip for users of managed Kubernetes solutions like GKE where it can be very cumbersome to get / configure / filter the Kubelet logs, but it is simple to view the container's logs (i.e. the logs of the entrypoint process). Commented Feb 2, 2023 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.