1

I'm developing my first App Engine Flexible Environment application.

The docs explain that virtual machines are restarted weekly:

VM instances are restarted on a weekly basis. During restarts Google's management services will apply any necessary operating system and security updates.

Will restarts result in downtime for apps with automatic scaling enabled? If so, are there any steps I can take to avoid downtime?

For example, I could frequently migrate traffic to new instances so that no instance runs for more than one week.

2
  • Did you find any better solution/workaround to handle such a situation?
    – Bala.Raj
    Mar 13, 2019 at 12:07
  • I did not, but I didn't investigate further. The scope of the project in question shrank to the point where there was no need for a GAE app. Mar 14, 2019 at 23:59

2 Answers 2

2

Well, Later I checked with the Google support team and here the recommendation from them to avoid the downtime.

My questions are:

  1. The weekly update is not fixed in time. Maybe there is a range in time in which I should expect the reboot of the instances? (ie: every Friday during the night).

  2. The weekly update involves all the instances, independently from when they were created? (ie: an instance created 1 hour or 1 day before the weekly update will be restarted?).

  3. How do we suppose to handle such a problem? it returns 502 for all request in the meantime.

1.- At this moment there is no way to know when the weekly restart is going to happen. GCP determine when is necessary and it does the restart of certain instances (once per week).

2.- No, as long as you have more than 1 one instance running you won’t see all of them being restarted at the same time.

3.- What we recommend to avoid downtime due to weekly restarts is having more than 1 instance as a minimum instance. Try to set at least 2 instances as a minimum.

I hope, this information is useful to others.

-1

The answer to your question is in the docs:

App Engine attempts to keep manual scaling instances running indefinitely, but there is no uptime guarantee. Hardware or software failures that cause early termination or frequent restarts can occur without warning and can take considerable time to resolve. Your application should be able to handle such failures.

Here are some good strategies for avoiding downtime due to instance restarts:

  • Use load balancing across multiple instances.
  • Configure more instances than required to handle normal traffic.
  • Write fall-back logic that uses cached results when a manual scaling instance is unavailable.
  • Reduce the amount of time it takes for your instances to start up and shutdown.
  • Duplicate the state information across more than one instance.
  • For long-running computations, checkpoint the state from time to time so you can resume it if it doesn't complete.
1
  • 1
    Good advice, but I'll use automatic scaling. Testing shows that one instance can handle normal traffic with spare resources. I hoped GAE would, with automatic scaling, add an instance before reducing running instances to zero. Does it? If not, I'll run with min_num_instances: 2. I could significantly reduce costs by setting min_num_instances: 1 often (e.g. 6 of 7 days). That is, if GAE provides a schedule (or advance notice) of weekly restarts. Does it? What is the maximum number of an app's instances that GAE will restart at once? Nov 6, 2017 at 1:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.