13

Sometimes we need to drain nodes in Kubernetes. When I manually set up a k8s cluster, I can drain the specific node then terminate that machine. While in EKS, nodes are under auto scaling group, which means I can't terminate a specific instance(node). If I manually terminate a instance, another instance(node) will be automatically added into eks cluster.

So is there any suggested method to drain a node in EKS?

3
  • What are you asking here? kubectl drain will drain a Node, ASG or not. Terminating an instance will effectively drain it, although not in the way you'd like. And yes, adding instances in place of other instances sounds like what an ASG was designed to do. If you don't want a replacement, perhaps scale the ASG down to prevent that behavior?
    – mdaniel
    Commented Aug 14, 2018 at 6:09
  • Currently if I want drain a node in EKS. I am going to: 1. run "kubectl drain xxx"; 2. terminate the ec2 instance drained in last step; 3. modify the desired instance number in ASG. While after these steps, ASG will terminate another instance(not the one i drained) and start a new instance. So after all, 2 instances are terminated and one new instance started, which is not what I want since I only want to drain & terminate one node(instance). Commented Sep 10, 2018 at 18:28
  • Well, if I'm hearing you correctly, you're conflating two separate things: you want node-1 drained and terminated, and the steps I outlined will 100% do that. But the other half of your question is somehow related to the sizing of the ASG (which is not related to draining anything). Please do consider updating your question to be more specific.
    – mdaniel
    Commented Sep 10, 2018 at 20:55

3 Answers 3

37

These steps should work:

  1. kubectl get nodes
  2. kubectl cordon <node name>
  3. kubectl drain <node name> --ignore-daemonsets or
    kubectl drain <node name> --ignore-daemonsets --delete-emptydir-data
  4. aws autoscaling terminate-instance-in-auto-scaling-group --instance-id <instance-id> --should-decrement-desired-capacity

For AWS autoscaling group, if you have nodes span out to multiple zones, consider delete nodes in each zones instead of all nodes from a single zone.

After the execution of the above commands, check the autoscaling group's desired number. It should decrease automatically. If you are using Terraform or another automation framework, don't forget to update your autoscaling group config in your infrastructure script.

2
  • 10
    In newer versions of K8s, the drain step will cordon the node as well
    – Andrew
    Commented Dec 16, 2019 at 13:25
  • If I'm using aws-ebs plugin for storing data will --delete-local-data loose me the data ? Commented Oct 21, 2022 at 12:18
2
1.) kubectl get nodes

2.) kubectl cordon <node name>

3.) kubectl drain <node name> --ignore-daemonsets --delete-emptydir-data

Flag --delete-local-data has been deprecated, This option is deprecated and will be deleted. Use --delete-emptydir-data.

0

For k8s < 1.23

kubectl get nodes
kubectl cordon "node-name"
kubectl drain "node-name" --ignore-daemonsets --delete-emptydir-data

For k8s >= 1.23 kubectl drain command already cordons the node.

kubectl get nodes
kubectl drain "node-name" --ignore-daemonsets --delete-emptydir-data

After draining the node, assuming you are using any Cluster Autoscaler(CA), your job is done, as the ca will detect when a node is unneeded and it will be removed.

If you do not, then you need to manually scale down the autoscaling node group of the drained node.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.