Any idea to view the log files of a crashed pod in kubernetes? My pod is listing it's state as "CrashLoopBackOff" after started the replicationController. I search the available docs and couldn't find any.
7 Answers
Assuming that your pod still exists:
kubectl logs <podname> --previous
$ kubectl logs -h
-p, --previous[=false]: If true, print the logs for the previous instance of the container in a pod if it exists.
-
9Not work for me:
Error from server (NotFound): pods "my-service" not found
– DherikCommented Oct 30, 2018 at 17:32 -
6
-
1Try
kubectl logs <podname> --namespace <some_namespace>
. If you don't know your namespace usekubectl get pods --all-namespaces
to find your pod and its namespace. Commented Apr 12, 2019 at 17:22 -
10
In many cases, kubectl logs <podname> --previous
is returning:
Error from server (BadRequest): previous terminated container "<container-name>" in pod "<pod-name>" not found
So you can try to check in the namespace's events (kubectl get events ..
) like @alltej showed.
If you don't find the reason for the error with kubectl logs / get events
and you can't view it with external logging tool I would suggest:
1 ) Check on which node that pod was running on with:
$kubectl get -n <namespace> pod <pod-name> -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName
NAME STATUS NODE
failed-pod-name Pending dns-of-node
(If you remove the <pod-name>
you can see other pods in the namespace).
2 ) SSH into that node and:
A ) Search for the failed pod container name in /var/log/containers/
and dump its .log
file and search for errors - in most of the cases the cause of error will be displayed there alongside with the actions / events that took place before the error.
B ) If previous step doesn't help try searching for latest System level errors by running:
sudo journalctl -u kubelet -n 100 --no-pager
.
-
1If you want to do an SSH into the node in AKS, you can do as follows:
kubectl debug node/aks-nodepool1-12345678-vmss000000 -it --image=mcr.microsoft.com/dotnet/runtime-deps:6.0
followed bychroot /host
, see learn.microsoft.com/en-us/azure/aks/node-access Commented Apr 5, 2022 at 18:03
kubectl logs
command only works if the pod is up and running. If they are not, you can use the kubectl events
command.
kubectl get events -n <your_app_namespace> --sort-by='.metadata.creationTimestamp'
By default it does not sort the events, hence the --sort-by
flag.
-
3This only lists the events, it does not provide logs about why there was a backoff Commented Sep 8, 2020 at 17:13
There was a bug in kubernetes that prevents logs obtaining for pods in CrashLoopBackOff state. Looks like it was fixed. Here issue on github with additional information
If the pod does not exist anymore:
kubectl describe pod {RUNTIME_NAME_OF_POD}
In the output you should have the section "Events" which contains the error messages that prevented the pod to start.
Container Failures could be due to resource limits reached:
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Wed, 18 Jan 2023 11:28:14 +0530
Finished: Wed, 18 Jan 2023 11:28:18 +0530
Ready: False
Restart Count: 13
OR
The application ended due to an error:
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Wed, 18 Jan 2023 11:50:59 +0530
Finished: Wed, 18 Jan 2023 11:51:03 +0530
Ready: False
Debugging container failure:
# Looking at pod status which will contain the above status information:
kubectl get pod POD_NAME -o yaml
# Watch the events to specific pod:
kubectl get events -w | grep POD_NAME_STRING
# For default container logs: Tailing the logs may give clue
kubectl logs -f POD_NAME
# For specific container: reason for application failure
kubectl logs -f POD_NAME --container CONTAINER_NAME
# only view logs of a crashed/restarted:
kubectl logs <POD_NAME> --previous
# Get logs of multiple pods with this label:
kubectl logs -l app=blue-green
# Looking at events:
kubectl describe deployment DEPLOYMENT_NAME
kubectl describe pod POD_NAME
As discussed on another StackOverflow question, I wrote an open source tool to do this
The main difference with the other answers is that this is triggered automatically when a pod crashes, so it can help avoid scenarios where you start debugging this much later on and the pod itself no longer exists and logs can't be fetched.