101

Any idea to view the log files of a crashed pod in kubernetes? My pod is listing it's state as "CrashLoopBackOff" after started the replicationController. I search the available docs and couldn't find any.

7 Answers 7

158

Assuming that your pod still exists:

kubectl logs <podname> --previous

$ kubectl logs -h
-p, --previous[=false]: If true, print the logs for the previous instance of the container in a pod if it exists.

4
  • 9
    Not work for me: Error from server (NotFound): pods "my-service" not found
    – Dherik
    Commented Oct 30, 2018 at 17:32
  • 6
    it returns nothing. Commented Jan 14, 2019 at 20:51
  • 1
    Try kubectl logs <podname> --namespace <some_namespace>. If you don't know your namespace use kubectl get pods --all-namespaces to find your pod and its namespace. Commented Apr 12, 2019 at 17:22
  • 10
    Is there way how to get logs from pre-previous pod?
    – Marosinho
    Commented Jan 14, 2021 at 8:44
21

In many cases, kubectl logs <podname> --previous is returning:

Error from server (BadRequest): previous terminated container "<container-name>" in pod "<pod-name>" not found

So you can try to check in the namespace's events (kubectl get events ..) like @alltej showed.

If you don't find the reason for the error with kubectl logs / get events and you can't view it with external logging tool I would suggest:

1 ) Check on which node that pod was running on with:

$kubectl get -n <namespace> pod <pod-name> -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName

NAME              STATUS    NODE
failed-pod-name   Pending   dns-of-node

(If you remove the <pod-name> you can see other pods in the namespace).

2 ) SSH into that node and:

A ) Search for the failed pod container name in /var/log/containers/ and dump its .log file and search for errors - in most of the cases the cause of error will be displayed there alongside with the actions / events that took place before the error.

B ) If previous step doesn't help try searching for latest System level errors by running:
sudo journalctl -u kubelet -n 100 --no-pager.

1
16

kubectl logs command only works if the pod is up and running. If they are not, you can use the kubectl events command.

kubectl get events -n <your_app_namespace> --sort-by='.metadata.creationTimestamp'

By default it does not sort the events, hence the --sort-by flag.

1
  • 3
    This only lists the events, it does not provide logs about why there was a backoff
    – 8bitjunkie
    Commented Sep 8, 2020 at 17:13
8

There was a bug in kubernetes that prevents logs obtaining for pods in CrashLoopBackOff state. Looks like it was fixed. Here issue on github with additional information

3

If the pod does not exist anymore:

kubectl describe pod {RUNTIME_NAME_OF_POD}

In the output you should have the section "Events" which contains the error messages that prevented the pod to start.

3

Container Failures could be due to resource limits reached:

State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Wed, 18 Jan 2023 11:28:14 +0530
  Finished:     Wed, 18 Jan 2023 11:28:18 +0530
Ready:          False
Restart Count:  13

OR

The application ended due to an error:

State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       Error
  Exit Code:    2
  Started:      Wed, 18 Jan 2023 11:50:59 +0530
  Finished:     Wed, 18 Jan 2023 11:51:03 +0530
Ready:          False

Debugging container failure:

# Looking at pod status which will contain the above status information:
kubectl get pod POD_NAME -o yaml

# Watch the events to specific pod:
kubectl get events -w | grep POD_NAME_STRING

# For default container logs: Tailing the logs may give clue
kubectl logs -f POD_NAME

# For specific container: reason for application failure
kubectl logs -f POD_NAME --container CONTAINER_NAME

# only view logs of a crashed/restarted: 
kubectl logs <POD_NAME> --previous

# Get logs of multiple pods with this label:
kubectl logs -l app=blue-green  


# Looking at events:
kubectl describe deployment DEPLOYMENT_NAME
kubectl describe pod POD_NAME
0

As discussed on another StackOverflow question, I wrote an open source tool to do this

enter image description here

The main difference with the other answers is that this is triggered automatically when a pod crashes, so it can help avoid scenarios where you start debugging this much later on and the pod itself no longer exists and logs can't be fetched.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.