6

I have created a google Kubernetes engine with autoscale enabled with minimum and maximum nodes. A few days ago I deployed couple of servers on production which increased the nodes count as expected. but when I deleted those deployments I expect it to resize the nodes which are to scale down. I waited more than an hour but it still did not scale down.

  • All my other pods are controlled by replica controller since I deployed with kind: deployment.
  • All my statefulset pods are using PVC as volume.

I'm not sure what prevented the nodes to scale down so I manually scaled the nodes for now. Since I made the changes manually I can not get the autoscaler logs now.

Does anyone know what could be the issue here?

GKE version is 1.16.15-gke.4300

As mentioned in this link https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node

  • I'm not using any local storage.
  • pods not having PodDisruptionBudget(don't know what is that)
  • Pods are created by deployments (helm charts)
  • only thing is I don't have "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" this annotation. is this must?
1
  • Is it possible to provide more details, like your YAML configurations like deployments, pvc, did you use in your configuration anything what is mentioned here ? What GKE version are you using? or maybe you hit some GKE autoscaler limitations
    – PjoterS
    Jan 19, 2021 at 10:47

2 Answers 2

6

I have tested Cluster Autoscaler on my GKE cluster. It work's bit differently than you expected.

Backgorund

You can enable autoscaling using command or enable it during creation like it's described in this documentation.

In Cluster Autoscaler documentation you can find various information like Operation criteria, Limitations, etc.

As I mentioned in comment section, Cluster Autoscaler - Frequently Asked Questions won't work if will encounter one of below situation:

Pods with restrictive PodDisruptionBudget.

Kube-system pods that:

  • are not run on the node by default, *
  • don't have a pod disruption budget set or their PDB is too restrictive (since CA 0.6).

Pods that are not backed by a controller object (so not created by deployment, replica set, job, statefulset etc). *

Pods with local storage. *

Pods that cannot be moved elsewhere due to various constraints (lack of resources, non-matching node selectors or affinity, matching anti-affinity, etc)

Pods that have the following annotation set: "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

For my tests I've used 6 nodes, with autoscaling range 1-6 and nginx application with requests cpu: 200m and memory: 128Mi.

As OP mentioned that is not able to provide autoscaler logs, I will paste my logs from Logs Explorer. Description of how they can be achieved is in Viewing cluster autoscaler events documentation.

In those logs you should search noScaleDown events. You will find there a few information, however the most important is:

reason: {
parameters: [
0: "kube-dns-66d6b7c877-hddgs"
]
messageId: "no.scale.down.node.pod.kube.system.unmovable"

As it's described in NoScaleDown node-level reasons for "no.scale.down.node.pod.kube.system.unmovable":

Pod is blocking scale down because it's a non-daemonset, non-mirrored, non-pdb-assigned kube-system pod. See the Kubernetes Cluster Autoscaler FAQ for more details.

Solution

If you want to make Cluster Autoscaler work on GKE, you have to create Disruptions with proper information, how to create it can be found in How to set PDBs to enable CA to move kube-system pods?

kubectl create poddisruptionbudget <pdb name> --namespace=kube-system --selector app=<app name> --max-unavailable 1

where you have to specify the correct selector and --max-unavailable or --min-available depends on your needs. For more details, please read Specifying a PodDisruptionBudget documentation.

Tests

$ kubectl get deploy,nodes
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   16/16   16           16          66m

NAME                                            STATUS   ROLES    AGE     VERSION
node/gke-cluster-1-default-pool-6d42fa0a-1ckn   Ready    <none>   11m     v1.16.15-gke.6000
node/gke-cluster-1-default-pool-6d42fa0a-2j4j   Ready    <none>   11m     v1.16.15-gke.6000
node/gke-cluster-1-default-pool-6d42fa0a-388n   Ready    <none>   3h33m   v1.16.15-gke.6000
node/gke-cluster-1-default-pool-6d42fa0a-5x35   Ready    <none>   3h33m   v1.16.15-gke.6000
node/gke-cluster-1-default-pool-6d42fa0a-pdfk   Ready    <none>   3h33m   v1.16.15-gke.6000
node/gke-cluster-1-default-pool-6d42fa0a-wqtm   Ready    <none>   11m     v1.16.15-gke.6000
$ kubectl get pdb -A
NAMESPACE     NAME      MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
kube-system   kubedns   1               N/A               1                     43m

Scaledown deployment

$ kubectl scale deploy nginx-deployment --replicas=2
deployment.apps/nginx-deployment scaled

After a while (~10-15 minutes) in the event viewer you will find the Decision event and inside you will find information that the node was deleted.

...
scaleDown: {
nodesToBeRemoved: [
0: {
node: {
mig: {
zone: "europe-west2-c"
nodepool: "default-pool"
name: "gke-cluster-1-default-pool-6d42fa0a-grp"
}
name: "gke-cluster-1-default-pool-6d42fa0a-wqtm"

Number of nodes decreased:

$ kubectl get nodes
NAME                                       STATUS   ROLES    AGE     VERSION
gke-cluster-1-default-pool-6d42fa0a-2j4j   Ready    <none>   30m     v1.16.15-gke.6000
gke-cluster-1-default-pool-6d42fa0a-388n   Ready    <none>   3h51m   v1.16.15-gke.6000
gke-cluster-1-default-pool-6d42fa0a-5x35   Ready    <none>   3h51m   v1.16.15-gke.6000
gke-cluster-1-default-pool-6d42fa0a-pdfk   Ready    <none>   3h51m   v1.16.15-gke.6000

Another place where you can confirm it's scaling down is kubectl get events --sort-by='.metadata.creationTimestamp'

Output:

5m16s       Normal    NodeNotReady                                                                                             node/gke-cluster-1-default-pool-6d42fa0a-wqtm   Node gke-cluster-1-default-pool-6d42fa0a-wqtm status is now: NodeNotReady
4m56s       Normal    NodeNotReady                                                                                             node/gke-cluster-1-default-pool-6d42fa0a-1ckn   Node gke-cluster-1-default-pool-6d42fa0a-1ckn status is now: NodeNotReady
4m          Normal    Deleting node gke-cluster-1-default-pool-6d42fa0a-wqtm because it does not exist in the cloud provider   node/gke-cluster-1-default-pool-6d42fa0a-wqtm   Node gke-cluster-1-default-pool-6d42fa0a-wqtm event: DeletingNode
3m55s       Normal    RemovingNode                                                                                             node/gke-cluster-1-default-pool-6d42fa0a-wqtm   Node gke-cluster-1-default-pool-6d42fa0a-wqtm event: Removing Node gke-cluster-1-default-pool-6d42fa0a-wqtm from Controller
3m50s       Normal    Deleting node gke-cluster-1-default-pool-6d42fa0a-1ckn because it does not exist in the cloud provider   node/gke-cluster-1-default-pool-6d42fa0a-1ckn   Node gke-cluster-1-default-pool-6d42fa0a-1ckn event: DeletingNode
3m45s       Normal    RemovingNode                                                                                             node/gke-cluster-1-default-pool-6d42fa0a-1ckn   Node gke-cluster-1-default-pool-6d42fa0a-1ckn event: Removing Node gke-cluster-1-default-pool-6d42fa0a-1ckn from Controller

Conclusion

By default, kube-system pods prevent CA from removing nodes on which they are running. Users can manually add PDBs for the kube-system pods that can be safely rescheduled elsewhere. It can be achieved using:

kubectl create poddisruptionbudget <pdb name> --namespace=kube-system --selector app=<app name> --max-unavailable 1

List of possible reasons why CA won't autoscale can be found in Cluster Autoscaler - Frequently Asked Questions.

To verify which pods could still block CA downscale, you can use Autoscaler Events.

9
  • what if we want to scale the node when the average utilization of other reaches 70% is it possible? Jan 20, 2021 at 14:49
  • what i have seen i am running single replicas of most of micro services due to some limitation but only 2-3 service are scalable with HPA. all service have resource limit and request set but when HPA scales those services either node crash due to memory pressure or pod evicted. Jan 20, 2021 at 14:51
  • @HarshManvar I think it would be better if you create new question with more details, your environment, your current config and your expectations.
    – PjoterS
    Jan 25, 2021 at 13:15
  • 1
    Do you know what I'm supposed to put for "app=<app name>"? Feb 16, 2022 at 12:20
0

I had the same error with Kubernetes Autoscaler. In my case it was Kube-dns pod which was blocking Autoscaler to shutdown the node. I tried creating PodDistributionBudget for Kube-dns pod but didn't fix the error.

Solution: Just add the annotation "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" to affected pod (in my case 'kube-dns' pod) by updating affected pod Yaml file.

To add this annotation by default to affected pod just update the template for that perticular pod and it will add this annotation to all existing affected pod and also create new affected pod with the annotation.

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.