I have an ASG with desired/min/max of 1/1/5 instances (I want ASG just for rolling deploys and zone failover). When I start the Instance refresh with MinHealthyPercentage=100,InstanceWarmup=180, the process starts by deregistration (the instance goes to draining mode almost immediately on my ALB, instead waiting the 180 Warmup seconds until the new instance is healthy) and the application becomes unavailable for a while. Note that this is not specific just to my case with one instance. If I had two instances, the process also starts by deregistering one of the instances and that does not fulfill the 100% MinHealthy constraint either (the app will stay available, though)! Is there any other configuration option I should tune to get the rolling update create and warm up the new instance first?
-
Did you ever get this resolved? I am having the same problem. This part is fine for me: It takes a set of instances out of service, terminates them, and launches a set of instances with the new desired configuration. However when it is then supposed to wait for instance to be healthy what ends up happening is the ASG moves on before it's actually healthy in the target group. This causes an outage for me.– RaptorPeteOct 24, 2022 at 21:27
3 Answers
Currently instance refresh always terminates before launching, and it uses the minHealthyPercent to determine batch size and when it can move on to the next batch.
It takes a set of instances out of service, terminates them, and launches a set of instances with the new desired configuration. Then, it waits until the instances pass your health checks and complete warmup before it moves on to replacing other instances.
...
Setting the minimum healthy percentage to 100 percent limits the rate of replacement to one instance at a time. In contrast, setting it to 0 percent causes all instances to be replaced at the same time.
https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-instance-refresh.html
-
1I came to the same conclusion after reading more of the docs. This is really annoying and if I was a conspiracy theorist I would say that Amazon is doing it that way to force us to keep more instances running :) I had to update my deploy job to scale out the app first, then do an instance refresh and scale down again, which involves more instance creations/terminations than I feel comfortable with but hey, it works.– KoubasAug 26, 2020 at 7:13
-
Glad its working :) I'd say probably just another victim of "minimum viable product" release schedules. Hopefully will be added later– ShahadAug 29, 2020 at 1:00
-
1I find this very confusing. If the default behavior is to first terminates the old ones and then start the new ones, how do keep a % of instances running to ensure uptime? Oct 10, 2020 at 16:21
-
1IMHO this is a bug and should at least be clearly documented by AWS. Mar 11, 2021 at 8:31
-
How do you manage to do a zero downtime deployment with ECS EC2 ? I can't find a way to create an extra instance during deployment. Jul 21, 2021 at 16:34
If you are running the 1 instance and using the Launch template with the Autoscaling it would be hard to rolling update the EC2 instance.
i am coming from the above scenario and hitting up on this immature feature of AWS.
it's mentioned in the limitation of instance refresh, it will scale down the instance and will recreate the new one instead of creating the first new one instance.
Instances terminated before launch: When there is only one instance in the Auto Scaling group, starting an instance refresh can result in an outage. This is because Amazon EC2 Auto Scaling terminates an instance and then launches a new instance.
Ref : https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-instance-refresh.html
i tried work around of scaling up the auto-scaling group desired size to 2, it will create a new instance with the latest AMI in the launch template.
Now you have two instances running the old version & latest version, you will be good to set the desired capacity now back to 1 in the auto-scaling group.
Auto-scaling desired capacity to 1 will delete the older instance and keep the latest instance with the latest AMI.
Command to update desired capacity to 2
- aws autoscaling update-auto-scaling-group --auto-scaling-group-name $ASG_GROUP --desired-capacity 2
Command to update desired capacity to 1
- aws autoscaling update-auto-scaling-group --auto-scaling-group-name $ASG_GROUP --desired-capacity 1
Instead of using the instance-refresh this worked well for me.
-
1This is the closest thing to the behaviour I would have expected from Instance Refresh, but it doesn't always kill the oldest instance for some reason. Feb 8 at 10:43
-
right, correct instance refresh dont gives a guarantee always to kill the oldest i wasted 2 days behind that. Mar 15 at 19:11
This does not seem to be the case anymore. An instance refresh creates now a fresh instance and terminates the old one after health checks are successful. AWS Support mentioned this behavior was not changed since 2020.
-
This is incorrect. It still (annoyingly) works exactly as OP stated.– ruoholaMar 16, 2022 at 11:23
-