I'm currently running a single node in a GKE node pool. If I want to test some new changes in parallel with the production pods, I manually scale up to a size of 2, and then spin up new pods. Once I'm done, I spin down the new pods.
So now I have 2 nodes, one running the production pods, and the other is only running the default GKE pods. Now I scale the node pool back down to 1 node.
I've noticed that sometimes the node containing my production pods is shut down so all the pods are migrated to the other node. This happens with no downtime, but it seems a bit odd.
I could hardly find any documentation about this behaviour, except for this one line from a Google how-to which says
The MIG does not differentiate between instances running Pods and instances without Pods. Resizing down removes instances at random.
Which seems to confirm what I'm seeing, but offers no further hints. Firstly as to why (couldn't it remove the instance with the least resources requested?), but more importantly as to whether there's any workaround.
I see that I can cordon and drain the node using kubectl
commands. If I do that, will the resize operation pick the cordoned node to remove, or is it still random, and I might end up with unschedulable pods?
If I've misunderstood any basic concept here please do point me in the right direction! Maybe I should be using an autoscaler on the node pool between 1 and 2 nodes and let it handle the rest? But most of the time I do not need 2, so I worry it will spin up and not 'notice' it can spin back down.