15

I've been reading the kubernetes documentation https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container

But it's still not clear for me what's the difference between spec.containers[].resources.limits.cpu and spec.containers[].resources.requests.cpu and what's the impact on the resource limitation

Can you please suggest some reads or books where this is explained in common english?

Thanks in advance

4 Answers 4

21

When Kubernetes pod is scheduled on a particular node it is required the pod to have enough resources to run. Kubernetes knows resources of it's node but how does kubernetes knows the how much resources will pod takes beforehand to schedule it effectively in nodes. For that requests will be used. When we specify a request of resource kubernetes will guarantee that pod will get that amount of resource.

On other hand limit limits the resource usage by a pod. Kubernetes will not allow a pod to take more resources than the limit. When it comes to CPU if you request more kubernetes will throttle pods CPU artificially. If pod exceed a limit pod will be it will be terminated. To make it simple it simple limit is always bigger than request.

This example will give you idea about request and limit. Think that there is a pod where you have specify its memory request as 7GB and memory limit as 10GB. There are three nodes in your cluster where node1 has 2GB of memory, node2 has 8GB memory and node3 has 16GB. Your pod will never be scheduled on node1. But it will either be sceduled on node2 or node3 depending on pod current memory usage. But if it is scheduled on node3 it will be terminated in any scenario it will exceed the memory usage of 10GB.

0
5

Memory is kind of trivial to understand. requests is guaranteed and limits is something that can not be exceeded. This also means that when you issue kubectl describe nodes | tail -10 for example, you could see a phrase like:

"Total limits may be over 100 percent, i.e., overcommitted".

This means that the total sum of requests.memory is <= 100% (otherwise pods could not be scheduled and this is the meaning of guaranteed memory). At the same time if you see a value that is higher then 100%, it means that the total sum of limits.memory can go above 100% (and this is the overcommitted part in the message). So when a node tries to schedule a pod, it will only check requests.memory to see if it has enough memory.

The cpu part if more complicated.

requests.cpu translates to cpu shares, and without looking at all pods on the node, it might make little to no sense to be honest. imho, the easiest way to understand this property is by looking at an example.

  • Suppose you have 100 cores available on a node, you deploy a single pod and set requests.cpu = 1000m. In such a case, your pod can use 100 cpus, bot min and max.

  • You have the same machine (100 cores), but you deploy two pods with requests.cpu = 1000m. In such a case, your pods can use 50 cores each minimum, and 100 max.

  • Same node, 4 pods (requests.cpu = 1000m). Each pod can use 25 cpu min, and 100 max.

You get the picture, it matters what all pods set for requests.cpu to get an overall picture.

limits.cpu is a lot more interesting and it translated to two properties on the cgroup : cpu period and cpu quota. It means how much time (quota) can you get in a certain timeframe (period). An example should make things more simple here aswell.

  • Suppose period=100ms and quota=20ms and you get a request that will finish in 50ms on your pod.

This is how it will look like:

|     100ms   ||     100ms   ||     100ms   |
| 20 ms ......|| 20 ms ......|| 10 ms ......|

Because it takes 50ms to process a request, and we have only 20ms available for every 100ms, it will take 300ms in total, to process our request.


That being said, there are quite a lot of people that recommend not setting the cpu, at all. google engineers, zalando, monzo, etc - including us. We do not do that, and there are strong reasons for that (that go beyond this question).

1
  • Could you clarify? If you have 100 cores available and try to start 2 pods with CPU requests set to 1000m, will one of them be never scheduled as the request cannot be satisfied by the node?
    – idetyp
    Mar 14 at 13:06
2

in short: for cpu & memory requests: k8s guarantee what you declared you will get when scheduler schedule your pods.

for cpu & memory limits: k8s guarantee you can not exceed the value you set.

the results when your pod exceed the limits:

  • for cpu: k8s throttling your container
  • for memory: OOM, k8s kill your pod
1
  • in general, there are a lot of advocates (from google, zalando, Monzo, etc) that do not advise to set the cpu, at all. Also, cpu and memory are set very differently on the host, and mean very different things.
    – Eugene
    Jan 5, 2022 at 9:46
2

Concept

  • Containers specify a request, which is the amount of that resource that the system will guarantee to the container
  • Containers specify a limit which is the maximum amount that the system will allow the container to use.

Best practices for CPU limits and requests on Kubernetes

  • Use CPU requests for everything and make sure they are accurate
  • Do NOT use CPU limits.

Best practices for Memory limits and requests on Kubernetes

  • Use memory limits and memory requests
  • Set memory limit = memory request

For more details on limits and request setting, please refer to this answer


Details

  • Containers can specify a resource request and limit, 0 <= request <= Node Allocatable & request <= limit <= Infinity
  • If a pod is successfully scheduled, the container is guaranteed the amount of resources requested. Scheduling is based on requests and not limits
  • The pods and its containers will not be allowed to exceed the specified limit. How the request and limit are enforced depends on whether the resource is compressible or incompressible
    • Compressible Resource Guarantees
      • Pods are guaranteed to get the amount of CPU they request, they may or may not get additional CPU time (depending on the other jobs running). This isn't fully guaranteed today because cpu isolation is at the container level. Pod level cgroups will be introduced soon to achieve this goal.
      • Excess CPU resources will be distributed based on the amount of CPU requested. For example, suppose container A requests for 600 milli CPUs, and container B requests for 300 milli CPUs. Suppose that both containers are trying to use as much CPU as they can. Then the extra 100 milli CPUs will be distributed to A and B in a 2:1 ratio (implementation discussed in later sections).
      • Pods will be throttled if they exceed their limit. If limit is unspecified, then the pods can use excess CPU when available.
    • Incompressible Resource Guarantees
      • Pods will get the amount of memory they request, if they exceed their memory request, they could be killed (if some other pod needs memory), but if pods consume less memory than requested, they will not be killed (except in cases where system tasks or daemons need more memory).
      • When Pods use more memory than their limit, a process that is using the most amount of memory, inside one of the pod's containers, will be killed by the kernel.

Purpose

  • Kubernetes provides different levels of Quality of Service to pods depending on what they request. Pods that need to stay up reliably can request guaranteed resources, while pods with less stringent requirements can use resources with weaker or no guarantee.

  • For each resource, we divide containers into 3 QoS classes: Guaranteed, Burstable, and Best-Effort, in decreasing order of priority. The relationship between "Requests and Limits" and "QoS Classes" is subtle.

    • If limits and optionally requests (not equal to 0) are set for all resources across all containers and they are equal, then the pod is classified as Guaranteed.
    • If requests and optionally limits are set (not equal to 0) for one or more resources across one or more containers, and they are not equal, then the pod is classified as Burstable. When limits are not specified, they default to the node capacity.
    • If requests and limits are not set for all of the resources, across all containers, then the pod is classified as Best-Effort.
  • Pods will not be killed if CPU guarantees cannot be met (for example if system tasks or daemons take up lots of CPU), they will be temporarily throttled.

  • Memory is an incompressible resource and so let's discuss the semantics of memory management a bit.

    • Best-Effort pods will be treated as lowest priority. Processes in these pods are the first to get killed if the system runs out of memory. These containers can use any amount of free memory in the node though.
    • Guaranteed pods are considered top-priority and are guaranteed to not be killed until they exceed their limits, or if the system is under memory pressure and there are no lower priority containers that can be evicted.
    • Burstable pods have some form of minimal resource guarantee, but can use more resources when available. Under system memory pressure, these containers are more likely to be killed once they exceed their requests and no Best-Effort pods exist.

Source: Resource Quality of Service in Kubernetes

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.