2

In our development environment, our functions app typically only requires a couple of servers.

Over the weekend we unexpectedly triggered a chain reaction which lead to gradual, sustained scaling out up to about 140 servers. Now that we've fixed the underlying issue and cleared the queues, the activity seems to have returned to normal. (Phew!)

The strange thing is that (30 mins later) we still have all those servers online. I would have expected them to start to get take offline quite quickly, but instead I'm seeing the number of "servers online" fluctuate (up as well as down) between about 110 and 140. The vast majority of these are sitting there with 0 requests/s, no CPU and no memory.

So, some questions:

  • Is this expected?
  • What are the heuristics that guide the number of servers that are brought / kept online?
  • When should I expect to see this number drop back down to the levels I'm used to?
2
  • As an aside, they did eventually calm down. I'm still interested in knowing about what controls this behaviour though.
    – Tom Wright
    Sep 3, 2018 at 3:59
  • 1
    You might find interesting a paper about AWS Lambda, Azure Functions and Google Cloud Functions, called Peeking Behind the Curtains of Serverless Platforms. My source here.
    – tgogos
    Sep 10, 2018 at 12:42

2 Answers 2

2

Just to add some to what @Mikhail wrote - how your function scale depends on the type of a trigger used. If you use a queue, the runtime will take a look at the queue length and scale up/down depending on the number of messages. With Event Hub this behaviour depends on the number of partition in a hub - the more you have, the more eager to scale your functions will be.

It is possible to take a look at the source code and try to understand at least part of the functionality. In fact, it is based on a timer and a concept of workers, which update their statuses and let the runtime decide, whether it is required to scale up or down.

The overall algorithm is described as follows:

protected virtual async Task MakeScaleDecision(string activityId, IWorkerInfo manager)
{
    if (DateTime.UtcNow < _scaleCheckUtc)
    {
        return;
    }

    try
    {
        var workers = await _table.ListNonStale();
        _tracer.TraceInformation(activityId, manager, workers.GetSummary("NonStale"));

        if (await TryRemoveIfMaxWorkers(activityId, workers, manager))
        {
            return;
        }

        if (await TryAddIfLoadFactorMaxWorker(activityId, workers, manager))
        {
            return;
        }

        if (await TrySwapIfLoadFactorMinWorker(activityId, workers, manager))
        {
            return;
        }

        if (await TryAddIfMaxBusyWorkerRatio(activityId, workers, manager))
        {
            return;
        }

        if (await TryRemoveIfMaxFreeWorkerRatio(activityId, workers, manager))
        {
            return;
        }

        if (await TryRemoveSlaveWorker(activityId, workers, manager))
        {
            return;
        }
    }
    catch (Exception ex)
    {
        _tracer.TraceError(activityId, manager, string.Format("MakeScaleDecision failed with {0}", ex));
    }
    finally
    {
        _scaleCheckUtc = DateTime.UtcNow.Add(_settings.ScaleCheckInterval);
    }
}

What is more, the answer why you see alive workers can be also found in the source code:

protected virtual async Task PingWorker(string activityId, IWorkerInfo worker)
{
    // if ping was unsuccessful, keep pinging.  this is to address
    // the issue where site continue to run on an unassigned worker.
    if (!_pingResult || _pingWorkerUtc < DateTime.UtcNow)
    {
        // if PingWorker throws, we will not update the worker status
        // this worker will be stale and eventually removed.
        _pingResult = await _eventHandler.PingWorker(activityId, worker);

        _pingWorkerUtc = DateTime.UtcNow.Add(_settings.WorkerPingInterval);
    }

    // check if worker is valid for the site
    if (_pingResult)
    {
        await _table.AddOrUpdate(worker);

        _tracer.TraceUpdateWorker(activityId, worker, string.Format("Worker loadfactor {0} updated", worker.LoadFactor));
    }
    else
    {
        _tracer.TraceWarning(activityId, worker, string.Format("Worker does not belong to the site."));

        await _table.Delete(worker);

        _tracer.TraceRemoveWorker(activityId, worker, "Worker removed");

        throw new InvalidOperationException("The worker does not belong to the site.");
    }
}

Unfortunately, some parts of the implementation are sealed(like IWorkerInfo), so you are unable to get the whole picture, we can only guess(or ask ;))

1
  • 1
    Great overview. In particular, we are using queues and it's interesting to know that the queue lengths will affect the scaling algo.
    – Tom Wright
    Sep 4, 2018 at 6:39
2

Scale Controller is a closed source proprietary piece of technology with undocumented behavior. There is no official answer to your question. The only way to answer it is to do what you did: run an experiment and measure. Such behavior may change over time though.

Functions do have some heuristics on when to scale out and when to scale in based on amount of evens and resource utilization. In the past, they were quite frugal but that did not provide good enough scale out pace. Now, they tend to err on the side of provisioning too many rather than too few.

The good news is that you shouldn't really care about scaling in, except for out of curiosity. You don't pay for idle servers.

1
  • 1
    Excellent closing note! You just saved me hours of research. You are absolutely right, we shouldn't care about idle servers as they are not billed.
    – GETah
    Feb 6, 2021 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.