22

I am trying to do a pip install from codeartifact from within a dockerbuild in aws codebuild.

This article does not quite solve my problem: https://docs.aws.amazon.com/codeartifact/latest/ug/using-python-packages-in-codebuild.html

The login to AWS CodeArtifct is in the prebuild; outside of the Docker context.

But my pip install is inside my Dockerfile (we pull from a private pypi registry).

How do I do this, without doing something horrible like setting an env variable to the password derived from reading ~/.config/pip.conf/ after running the login command in prebuild?

0

2 Answers 2

19

You can use the environment variable: PIP_INDEX_URL[1].

Below is an AWS CodeBuild buildspec.yml file where we construct the PIP_INDEX_URL for CodeArtifact by using this example from the AWS documentation.

buildspec.yml

  pre_build:
    commands:
      - echo Getting CodeArtifact authorization...
      - export CODEARTIFACT_AUTH_TOKEN=$(aws codeartifact get-authorization-token --domain "${CODEARTIFACT_DOMAIN}" --domain-owner "${AWS_ACCOUNT_ID}" --query authorizationToken --output text)
      - export PIP_INDEX_URL="https://aws:${CODEARTIFACT_AUTH_TOKEN}@${CODEARTIFACT_DOMAIN}-${AWS_ACCOUNT_ID}.d.codeartifact.${AWS_DEFAULT_REGION}.amazonaws.com/pypi/${CODEARTIFACT_REPO}/simple/"

In your Dockerfile, add an ARG PIP_INDEX_URL line just above your RUN pip install -r requirements.txt so it can become an environment variable during the build process:

Dockerfile

# this needs to be added before your pip install line!
ARG PIP_INDEX_URL

RUN pip install -r requirements.txt

Finally, we build the image with the PIP_INDEX_URL build-arg.

buildspec.yml

  build:
    commands:
      - echo Building the Docker image...
      - docker build -t "${IMAGE_REPO_NAME}" --build-arg PIP_INDEX_URL .

As an aside, adding ARG PIP_INDEX_URL to your Dockerfile shouldn't break any existing CI or workflows. If --build-arg PIP_INDEX_URL is omitted when building an image, pip will still use the default PyPI index.

Specifying --build-arg PIP_INDEX_URL=${PIP_INDEX_URL} is valid, but unnecessary. Specifying the argument name with no value will make Docker take its value from the environment variable of the same name[2].

Security note: If someone runs docker history ${IMAGE_REPO_NAME}, they can see the value of ${PIP_INDEX_URL}[3] . The token is only good for a maximum of 12 hours though, and you can shorten it to as little as 15 minutes with the --duration-seconds parameter of aws codeartifact get-authorization-token[4], so maybe that's acceptable. If your Dockerfile is a multi-stage build, then it shouldn't be an issue if you're not using ARG PIP_INDEX_URL in your target stage. docker build --secret does not seem to be supported in CodeBuild at this time.

3
  • 1
    thanks for this! I will investigate switching to this methodology
    – Tommy
    Commented Aug 3, 2021 at 10:44
  • 1
    Will docker cache the stages after ARG PIP_INDEX_URL after the token refreshes ? If not is there a workaround ?
    – Roy Assis
    Commented Oct 20, 2022 at 6:10
  • 1
    @Roy If the value of PIP_INDEX_URL changes, I'm pretty sure the cache is invalidated starting at the line that the PIP_INDEX_URL is first referenced. To get around this, you could create some kind of builder image in a separate Dockerfile and push that image to an image repository. Then refer to that in your final image's Dockerfile using FROM myrepo/mybuilder as builder or COPY --from=myrepo/mybuilder /thing.py /app/thing.py. You would then manually recreate your builder image only when your requirements.txt changes instead of every time your PIP_INDEX_URL changed.
    – Phistrom
    Commented Oct 24, 2022 at 16:55
5

So, here is how I solved this for now. Seems kinda hacky, but it works. (EDIT: we have since switched to @phistrom answer)

  1. In the prebuild, I run the command and copy ~/.config/pip/pip.conf to the current build directory:
pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      ...
      - echo Fetching pip.conf for PYPI
      - aws codeartifact --region us-east-1 login --tool pip --repository ....
      - cp ~/.config/pip/pip.conf .
  build:
    commands:
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
  1. Then in the Dockerfile, I COPY that file in, do the pip install, then rm it
COPY requirements.txt pkg/
COPY --chown=myuser:myuser pip.conf /home/myuser/.config/pip/pip.conf
RUN pip install -r ./pkg/requirements.txt
RUN pip install ./pkg
RUN rm /home/myuser/.config/pip/pip.conf

2
  • Using this method your pip.conf (and the credentials it contains) will still be contained in the docker image because of the way the layered file system in docker images works, which is not best practice. Commented Oct 29, 2021 at 10:31
  • @Snorfalorpagus that is true; however it is only good for 12 hours (the file will be invalid then). But i agree, and we have switched to using the other answer here (which came after this)
    – Tommy
    Commented Oct 29, 2021 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.