7

If I understand correctly the conditions for Kubernetes to OOM kill a pod or container (from komodor.com):

If a container uses more memory than its memory limit, it is terminated with an OOMKilled status. Similarly, if overall memory usage on all containers, or all pods on the node, exceeds the defined limit, one or more pods may be terminated.

This means that if a container in the pod exceeds the total memory it will be killed (the container) but not the pod itself. Similarly, if there are multiple containers in a pod and the pod itself exceeds its memory limitation, which is the sum of memory limits of all the containers in that pod - the pod will be OOM killed. However, the latter only seems possibly if one of the containers exceeds its memory allowance. In this case - wouldn't the container be killed first?

I'm trying to understand the actual conditions in which a pod is OOM killed instead of a container.

I've also noticed that when there is one container in the pod and that container is exceeding its memory allowance repeatedly - the pod and container are killed intermittently. I observed this - the container would restart, which would be observable by watching the logs from the pod, and every second time - the pod is killed and restarted, incrementing its restart count.

If it helps to understand the behavior - the QOS class of the pod is Burstable.

2
  • How do you set the memory constraints on a pod? Commented Oct 25, 2022 at 15:28
  • @SoftwareEngineer there is only one container per pod and for that container I set resources->limits->memory: 512Mi and resources->requests->memory: 256Mi.
    – Alechko
    Commented Oct 26, 2022 at 13:03

1 Answer 1

20

Pods aren't OOM killed at all. OOMKilled is a status ultimately caused by a kernel process (OOM Killer) that kills processes (containers are processes), which is then recognised by the kubelet which sets the status on the container. If the main container in a pod is killed then by default the pod will be restarted by the kubelet. A pod cannot be terminated, because a pod is a data structure rather than a process. Similarly, it cannot have a memory (or CPU) limit itself, rather it is limited by the sum of its component parts.

The article you reference uses imprecise language and I think this is causing some confusion. There is a better, shorter, article on medium that covers this more accurately, and a longer and much more in depth article here.

6
  • That's very insightful. So if the process going OOM is the main container in the pod - only then is the pod restarting? Could it be related to the fact that my pod is running with pid different than 1? (not using 'exec': petermalmgren.com/signal-handling-docker)
    – Alechko
    Commented Oct 26, 2022 at 14:44
  • Could be, though pid 1 isn't always necessary. Even basic docker supports an --init flag that's going to force your main process away from pid 1. And, your pid in the container doesn't matter because you'll have a different pid in the kernal. Commented Oct 26, 2022 at 14:48
  • Btw, I have a pod with a sidecar that regularly fails with an OOMKilled status because it has a memory leak (which is in a 3rd party library). This suits me fine. I have other replicas running, so the only side effect is to slightly reduce my throughput while the pod automatically restarts. I achieve this by linking the pod's health to the running sidecar as well as the main container (there's another container too, acting as a network proxy, but it's never problematic). Commented Oct 26, 2022 at 14:55
  • A follow-up question: According to this response, there could be a strange condition, in which a secondary container is killed, but the pod continues to live. It seems problematic to me, as most likely the pod is not functioning as expected, and I would have expected Kubernetes to restart it. Why isn't it so?
    – Nadav
    Commented Jun 7, 2023 at 7:22
  • @Nadav some sidecars, init containers for example, are designed to die while the pod lives on, so the situation you describe isn't actually very strange at all. Kubernetes can't distinguish a 'mission critical' container unless you attach it to the pod health metric. You should do this for all sidecars that are tied to the pod's lifetime. Commented Jun 7, 2023 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.