2

I'm currently running a single node in a GKE node pool. If I want to test some new changes in parallel with the production pods, I manually scale up to a size of 2, and then spin up new pods. Once I'm done, I spin down the new pods.

So now I have 2 nodes, one running the production pods, and the other is only running the default GKE pods. Now I scale the node pool back down to 1 node.

I've noticed that sometimes the node containing my production pods is shut down so all the pods are migrated to the other node. This happens with no downtime, but it seems a bit odd.

I could hardly find any documentation about this behaviour, except for this one line from a Google how-to which says

The MIG does not differentiate between instances running Pods and instances without Pods. Resizing down removes instances at random.

Which seems to confirm what I'm seeing, but offers no further hints. Firstly as to why (couldn't it remove the instance with the least resources requested?), but more importantly as to whether there's any workaround.

I see that I can cordon and drain the node using kubectl commands. If I do that, will the resize operation pick the cordoned node to remove, or is it still random, and I might end up with unschedulable pods?

If I've misunderstood any basic concept here please do point me in the right direction! Maybe I should be using an autoscaler on the node pool between 1 and 2 nodes and let it handle the rest? But most of the time I do not need 2, so I worry it will spin up and not 'notice' it can spin back down.

1 Answer 1

2

Sounds like you already were thinking of this. One potential solution could be to enable cluster autoscaler and then the downsizing would automatically happen for the nodes that are under utilized. In addition, you could also explicitly cordon and/or drain a node and then autoscaler would automatically remove that node pretty soon.

Note you don't have to cordon or drain when you have autoscaler enabled. Cluster autoscaler will automatically remove nodes that are under utilized if there is plenty capacity available on other nodes.

The following doc will be helpful to read through: https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler Especially read the part about the profiles. If your workload can tolerate optimize utilization profile that might be what you're looking for to get more aggressive downsizing.

2
  • 1
    Thanks for the advice. I tried enabling an autoscaler on the node pools, and after struggling with auto-provisioning for a bit (which did not play nicely with GPUs at all), it worked quite nicely. Seems like the best approach even if it's a little disconcerting to leave it all to an automated process instead of having that manual control! (Particularly leaving an extra GPU node running as you leave work for the weekend...) May 20, 2022 at 16:05
  • @AndrewChinery you can still manually cordon and drain nodes as you see fit. You can also manually adjust the current size of the nodepool but not sure what the effect of that will be. May 20, 2022 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.