24

Will the cold starts of my AWS Lambda function take longer if I use a image from ECR instead of a jar from S3 as my source code? I'm thinking that yes, because the image is larger due to the additional OS layer (even though... the regular Lambda should have some OS layer as well), but I couldn't find any performance benchmarks.

Thanks!

6 Answers 6

17

I'm surprised at the conclusion of the other answers here.

IT DEPENDS

Linked earlier, this blog post performs data tests. From that post:

enter image description here enter image description here

If your function is a pure function. It'll perform way better (1st picture) as most have said, but once your function acts more like a Framework the size of the zip grows and all it takes is a few megs and S3 is simply too slow.

To be clear, your container/program needs to have a fast start time, but that's irrespective of the size or lambda.

That second graph is incredible, 5GB container loading in < 2s.

13
  • 1
    The above charts are basically wrong. Lambda uses lazy loading for container storage. If you don't touch every byte of that 5GB, then it's not actually loaded, and you'll see the same load times as for a tiny container. What's more, Lambda has multiple levels of cache, which makes it almost impossible to properly test cold start times. If you're doing repeated testing of even similar containers, even your cold starts will be pulling from a nearby cache, rather than going all the way to S3
    – jameslol
    Commented Feb 7 at 23:04
  • 1
    Interesting insight, but no, i think it's misleading at best to say that huge containers load fast, without stating that this is only if you don't use any of it. If you're loading 5gb containers and using so little of them that they're loading in 1 second, the take-home message is that you're doing something weird, not "it's so quick"
    – jameslol
    Commented Feb 9 at 6:51
  • 1
    Sorry if my bluntness got us off on the wrong foot. What I'm trying to say is that the start time is actually not irrespective of size or lambda. I have containers which initialise very quickly on my local machine (200ms) and very slowly on Lambda (2-4 seconds). This is because every file (block) in every library that is lazy-loaded has to be pulled using a separate call to S3. A process which is normally CPU-bound is now suddenly very much IO-bound.
    – jameslol
    Commented Feb 14 at 5:48
  • 1
    What's worse, the testing methodology for the two charts above are totally different. In the first chart, a heap of NPM libraries are initialised, which is why it takes 4 seconds for 35MB. (No way Lamba is spending 4 seconds on the IO to download 35MB from S3.) Whereas in the second chart, the author states that the container is just filled with random text files. So the container test was actually doing less work than the non-container test, which as displayed here falsely gives the impression that once your package size gets to 35MB or more, you should switch to a container
    – jameslol
    Commented Feb 14 at 5:55
  • 1
    @jameslol is correct; that graph is misleading to the point of being plain wrong. I have a docker image that is 60MB, and it takes 1.997 seconds just to import boto3, which is measured via PYTHONPROFILEIMPORTTIME=1
    – sam2426679
    Commented Jun 29 at 21:39
8

Yes, you will have longer cold starts, resulting in much higher response times.

It is really dependent on the image you're downloading from ECR but in general, it will be slower as a Docker container is being used instead of Lambda managing the runtime environment for you (which reduces the time to it takes to start a new execution environment for cold Lambdas).

The main cause is the size of the ECR image, which is also why there is a limit on large Lambda ZIP archives.

You can see here how the size will affect the 2 tasks that run during cold start, defined by AWS as Download your code & Start new execution environment.

cold start duration tasks & invocation duration tasks

I would advise you to definitely use the managed runtime as opposed to containers unless you need to use them as it will automatically result in faster execution.

3
  • 6
    This makes sense from a theoretical standpoint (at least as far as cold starts go), but do you have any evidence to back the "much higher latency" claim?
    – dskrvk
    Commented Jan 21, 2022 at 23:47
  • 1
    @dskrvk I've rephrased to clarify what I meant - is that better? Commented Feb 10, 2022 at 12:00
  • 1
    Note that with latest optimizations for containerized Lambda, currently it may actually be the other way around: aaronstuyvenberg.com/posts/containers-on-lambda
    – qalis
    Commented Mar 4 at 21:32
8

"Lambda also optimizes the image and caches it close to where the functions runs so cold start times are the same as for .zip archives" from https://aws.amazon.com/blogs/compute/working-with-lambda-layers-and-extensions-in-container-images/

5

Docker images will be definitely slower cold start. This is because of bigger size and additional OS. Lambda loads your code into its managed environment instead loading you whole docker image.

Links for reference: Some graphs comparing docker with native java lambda: https://mikhail.io/serverless/coldstarts/aws/

Additional information about lambda containers: https://chariotsolutions.com/blog/post/getting-started-with-lambda-container-images/

0

Yes, packaging a lambda as a container image rather than a zip will lead to longer cold starts.

There are 2 reasons for this:

  1. Container images tend to be larger, thus taking more time to load.
  2. Containers need to initialize the underlying operating system.

In general, use a container over a plain lambda when you benefit from having the underlying OS functionality. Otherwise, stick to zip files.

0

The code included here is helpful for converting your docker-based lambda to a zip-based lambda.

In our case, we needed an entrypoint lambda to handle incoming requests from API gateway, and these requests needed to be answered as quickly as possible. The entrypoint is just responsible for validating the payload and then pushing the work to SQS for async resolution.

We need the entrypoint Lambda to be a zip deployment so that it responds asap, but then we use the docker deployments for the async handlers that pull work from SQS.

This way, we build the framework via a single Dockerfile, and then we simply dump the framework from the built docker image to a .zip file for the entrypoint Lambda.

(We previously experimented with deploying the entrypoint Lambda as a docker container but found that it was too slow. The details of this experiment are in this comment.)


The directory structure looks like:

project_root_dir
├── framework_lib_main.Dockerfile
├── py-lib
│   └── framework_lib_main
│       ├── *.py
│   └── framework_lib_a
│       ├── *.py
│   └── framework_lib_b
│       ├── *.py

The framework_lib_main.Dockerfile looks like:

# https://docs.aws.amazon.com/lambda/latest/dg/python-image.html#python-alt-create

# Use amazonlinux image to support extracting to zip for lambda:
# https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-source-dist

##
# This will build `framework_lib_main`
#
# The requirements.txt of `framework_lib_main` includes local packages:
#   - framework_lib_a
#   - framework_lib_b
# Each local package has its own setup.py
#
# The requirements.txt of `framework_lib_main` also includes non-local packages, for example:
#   - boto3
#   - awslambdaric
##

ARG PY_VERSION="3.11"
ARG BUILD_DIR="/py-lib"

FROM public.ecr.aws/amazonlinux/amazonlinux:latest AS build-image
ARG PY_VERSION
ARG BUILD_DIR
RUN mkdir -p ${BUILD_DIR}

WORKDIR /py-lib-src
COPY ./framework_lib_a ./framework_lib_a/
COPY ./framework_lib_b ./framework_lib_b/

RUN dnf update && dnf install -y python${PY_VERSION} python${PY_VERSION}-pip

WORKDIR /py-lib-src/framework_lib_main
COPY ./framework_lib_main/requirements.txt .
RUN \
  /usr/bin/python${PY_VERSION} -m pip install --upgrade --no-cache-dir pip && \
  /usr/bin/python${PY_VERSION} -m pip install --upgrade --no-cache-dir pip-tools && \
  /usr/bin/python${PY_VERSION} -m pip install --no-cache-dir --target "${BUILD_DIR}" -r requirements.txt

COPY ./framework_lib_main .
RUN /usr/bin/python${PY_VERSION} -m pip install --no-cache-dir --target "${BUILD_DIR}" .

FROM public.ecr.aws/amazonlinux/amazonlinux:latest
ARG PY_VERSION
ARG BUILD_DIR

RUN dnf update && dnf install -y python${PY_VERSION}

WORKDIR ${BUILD_DIR}
COPY --from=build-image ${BUILD_DIR} ${BUILD_DIR}

RUN ln -s /usr/bin/python${PY_VERSION} /usr/local/bin/python_entrypoint
ENTRYPOINT ["/usr/local/bin/python_entrypoint"]

Below is the script responsible for (1) creating the .zip, (2) uploading the .zip to S3, and (3) updating the Lambda to use the new .zip:

if [[ -z "$ECR_REPO_URI" ]]; then
  echo "no value for \$ECR_REPO_URI" >&2
  exit 1
fi

if [[ -z "$S3_BUCKET" ]]; then
  echo "no value for \$S3_BUCKET" >&2
  exit 1
fi

if [[ -z "$S3_KEY" ]]; then
  echo "no value for \$S3_KEY" >&2
  exit 1
fi

if [[ -z "$FUNCTION_NAME" ]]; then
  echo "no value for \$FUNCTION_NAME" >&2
  exit 1
fi


push_to_s3() {
  ZIP_DIR='/tmp/docker_to_lambda'
  ZIP_PATH="$ZIP_DIR/lambda.zip"
  SRC_DIR='/py-lib'

  mkdir -p "$ZIP_DIR"

  docker run \
    --rm \
    -v "$ZIP_DIR:$ZIP_DIR" \
    --entrypoint /bin/sh \
    "$ECR_REPO_URI:latest" \
    -c "$(cat <<-EOF
      dnf update && \
      dnf install -y findutils zip && \
      cd "$SRC_DIR" && \
      chmod -R 644 . && \
      find .            -type d -exec chmod 755 '{}' \; && \
      find . -perm /111 -type f -exec chmod 755 '{}' \; && \
      zip -9 -r "$ZIP_PATH" *
    EOF
    )"

  aws s3api put-object \
    --bucket "$S3_BUCKET" \
    --key "$S3_KEY" \
    --body "$ZIP_PATH"

  rm "$ZIP_PATH"
}


update_lambda() {
  (aws lambda update-function-code \
    --function-name "$FUNCTION_NAME" \
    --s3-bucket "$S3_BUCKET" \
    --s3-key "$S3_KEY" \
    1>/dev/null) || return 1
  aws lambda wait function-updated --function-name "$FUNCTION_NAME" || return 1
}


push_to_s3 && update_lambda
1
  • 1
    Thanks for sharing. It'll take me a while to get to this as it isn't a runnable thing.
    – nitsujri
    Commented Jul 29 at 1:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.