2

I have a node project in GitLab. I have CI setup to build Docker image and upload it to AWS ECR. Which was working fine 3 days ago.

.gitlab-ci.yml

image: docker:latest

services:
  - docker:dind

stages:
  - build

variables:
  REPOSITORY_URL: 708604513256.dkr.ecr.ap-south-1.amazonaws.com/foo-bar/api:latest

before_script:
  - apk add --no-cache curl jq python py-pip
  - pip install awscli

build:
  stage: build
  only:
    - master
  script:
    - $(aws ecr get-login --no-include-email --region ap-south-1)
    - docker build -t $REPOSITORY_URL .
    - docker push $REPOSITORY_URL

Log

Running with gitlab-runner 13.0.0 (c127439c)
   on docker-auto-scale fa6cab46
Preparing the "docker+machine" executor
 Using Docker executor with image docker:latest ...
 Starting service docker:dind ...
 Pulling docker image docker:dind ...
 Using docker image sha256:ded0f436a6a380f373f1840de86b6887cff86600946490b3a44a890bc3d95657 for docker:dind ...
 Waiting for services to be up and running...
 *** WARNING: Service runner-fa6cab46-project-17319365-concurrent-0-f592aab325d1bb8d-docker-0 probably didn't start properly.
 Health check error:
 service "runner-fa6cab46-project-17319365-concurrent-0-f592aab325d1bb8d-docker-0-wait-for-service" timeout
 Health check container logs:
 Service container logs:
 2020-06-03T12:35:33.145705974Z time="2020-06-03T12:35:33.145584090Z" level=info msg="Starting up"
 2020-06-03T12:35:33.147344480Z time="2020-06-03T12:35:33.147268652Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
 2020-06-03T12:35:33.147688820Z time="2020-06-03T12:35:33.147629824Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
 2020-06-03T12:35:33.148653260Z time="2020-06-03T12:35:33.148586467Z" level=info msg="libcontainerd: started new containerd process" pid=18
 2020-06-03T12:35:33.148776016Z time="2020-06-03T12:35:33.148727154Z" level=info msg="parsed scheme: \"unix\"" module=grpc
 2020-06-03T12:35:33.148826946Z time="2020-06-03T12:35:33.148797744Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
 2020-06-03T12:35:33.148911802Z time="2020-06-03T12:35:33.148873819Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
 2020-06-03T12:35:33.148958763Z time="2020-06-03T12:35:33.148932476Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
 2020-06-03T12:35:33.200268502Z time="2020-06-03T12:35:33.200146030Z" level=info msg="starting containerd" revision=7ad184331fa3e55e52b890ea95e65ba581ae3429 version=v1.2.13 
 2020-06-03T12:35:33.200692597Z time="2020-06-03T12:35:33.200637628Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1 
 2020-06-03T12:35:33.200867133Z time="2020-06-03T12:35:33.200823919Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1 
 2020-06-03T12:35:33.201154379Z time="2020-06-03T12:35:33.201104050Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" 
 2020-06-03T12:35:33.201216139Z time="2020-06-03T12:35:33.201181022Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1 
 2020-06-03T12:35:33.205676519Z time="2020-06-03T12:35:33.205599239Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "ip: can't find device 'aufs'\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n": exit status 1" 
 2020-06-03T12:35:33.205881591Z time="2020-06-03T12:35:33.205836399Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1 
 2020-06-03T12:35:33.206055158Z time="2020-06-03T12:35:33.206013249Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1 
 2020-06-03T12:35:33.206518393Z time="2020-06-03T12:35:33.206440358Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1 
 2020-06-03T12:35:33.206807173Z time="2020-06-03T12:35:33.206761104Z" level=info msg="skip loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1 
 2020-06-03T12:35:33.206870377Z time="2020-06-03T12:35:33.206833950Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1 
 2020-06-03T12:35:33.206991282Z time="2020-06-03T12:35:33.206951423Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" 
 2020-06-03T12:35:33.207038857Z time="2020-06-03T12:35:33.207012167Z" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "ip: can't find device 'aufs'\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n": exit status 1" 
 2020-06-03T12:35:33.207105105Z time="2020-06-03T12:35:33.207071463Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" 
 2020-06-03T12:35:33.214130639Z time="2020-06-03T12:35:33.214040730Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1 
 2020-06-03T12:35:33.214248026Z time="2020-06-03T12:35:33.214196991Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1 
 2020-06-03T12:35:33.228942452Z time="2020-06-03T12:35:33.221454702Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.228984856Z time="2020-06-03T12:35:33.221493565Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.228990005Z time="2020-06-03T12:35:33.221505991Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.228993696Z time="2020-06-03T12:35:33.221519285Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.228997084Z time="2020-06-03T12:35:33.221531861Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.229000512Z time="2020-06-03T12:35:33.221542620Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.229003761Z time="2020-06-03T12:35:33.221553454Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.229006982Z time="2020-06-03T12:35:33.221564536Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1 
 2020-06-03T12:35:33.229010338Z time="2020-06-03T12:35:33.221759106Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2 
 2020-06-03T12:35:33.229013595Z time="2020-06-03T12:35:33.221856820Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1 
 2020-06-03T12:35:33.229016741Z time="2020-06-03T12:35:33.222190164Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1 
 2020-06-03T12:35:33.229019947Z time="2020-06-03T12:35:33.222213414Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1 
 2020-06-03T12:35:33.229032207Z time="2020-06-03T12:35:33.225029858Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229035721Z time="2020-06-03T12:35:33.225067184Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229052495Z time="2020-06-03T12:35:33.225080743Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229056388Z time="2020-06-03T12:35:33.225092135Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229059541Z time="2020-06-03T12:35:33.225103233Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229062675Z time="2020-06-03T12:35:33.225113980Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229065735Z time="2020-06-03T12:35:33.225123860Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229068798Z time="2020-06-03T12:35:33.225163545Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229071960Z time="2020-06-03T12:35:33.225174404Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1 
 2020-06-03T12:35:33.229075191Z time="2020-06-03T12:35:33.225684688Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229078297Z time="2020-06-03T12:35:33.225716450Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229081446Z time="2020-06-03T12:35:33.225727678Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229084533Z time="2020-06-03T12:35:33.225738133Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1 
 2020-06-03T12:35:33.229087706Z time="2020-06-03T12:35:33.225983911Z" level=info msg=serving... address="/var/run/docker/containerd/containerd-debug.sock" 
 2020-06-03T12:35:33.229090873Z time="2020-06-03T12:35:33.226055956Z" level=info msg=serving... address="/var/run/docker/containerd/containerd.sock" 
 2020-06-03T12:35:33.229094308Z time="2020-06-03T12:35:33.226065898Z" level=info msg="containerd successfully booted in 0.026443s" 
 2020-06-03T12:35:33.243372181Z time="2020-06-03T12:35:33.240058390Z" level=info msg="Setting the storage driver from the $DOCKER_DRIVER environment variable (overlay2)"
 2020-06-03T12:35:33.243390431Z time="2020-06-03T12:35:33.240296002Z" level=info msg="parsed scheme: \"unix\"" module=grpc
 2020-06-03T12:35:33.243394742Z time="2020-06-03T12:35:33.240311103Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
 2020-06-03T12:35:33.243398341Z time="2020-06-03T12:35:33.240336947Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
 2020-06-03T12:35:33.243411874Z time="2020-06-03T12:35:33.240345801Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
 2020-06-03T12:35:33.281680505Z time="2020-06-03T12:35:33.260046911Z" level=info msg="parsed scheme: \"unix\"" module=grpc
 2020-06-03T12:35:33.281700448Z time="2020-06-03T12:35:33.260069151Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
 2020-06-03T12:35:33.281705205Z time="2020-06-03T12:35:33.260086077Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
 2020-06-03T12:35:33.281709991Z time="2020-06-03T12:35:33.260095568Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
 2020-06-03T12:35:33.302342647Z time="2020-06-03T12:35:33.298907986Z" level=info msg="Loading containers: start."
 2020-06-03T12:35:33.356170350Z time="2020-06-03T12:35:33.346872010Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'\nbridge                167936  1 br_netfilter\nstp                    16384  1 bridge\nllc                    16384  2 bridge,stp\nip: can't find device 'br_netfilter'\nbr_netfilter           24576  0 \nbridge                167936  1 br_netfilter\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n, error: exit status 1"
 2020-06-03T12:35:33.459014885Z time="2020-06-03T12:35:33.458902627Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address"
 2020-06-03T12:35:33.512323955Z time="2020-06-03T12:35:33.511972176Z" level=info msg="Loading containers: done."
 2020-06-03T12:35:33.538197317Z time="2020-06-03T12:35:33.537823654Z" level=info msg="Docker daemon" commit=42e35e61f3 graphdriver(s)=overlay2 version=19.03.11
 2020-06-03T12:35:33.538256362Z time="2020-06-03T12:35:33.537959192Z" level=info msg="Daemon has completed initialization"
 2020-06-03T12:35:33.597277606Z time="2020-06-03T12:35:33.593481285Z" level=info msg="API listen on [::]:2375"
 2020-06-03T12:35:33.597295730Z time="2020-06-03T12:35:33.593560537Z" level=info msg="API listen on /var/run/docker.sock"
 *********
 Pulling docker image docker:latest ...
 Using docker image sha256:0bfe00e7bcd536451161fd64b2fe05f4ff6373ce9dd241e9762b68c7f56ce438 for docker:latest ...
Preparing environment
 Running on runner-fa6cab46-project-17319365-concurrent-0 via runner-fa6cab46-srm-1591186323-a8bad638...
Getting source from Git repository
 $ eval "$CI_PRE_CLONE_SCRIPT"
 Fetching changes with git depth set to 50...
 Initialized empty Git repository in /builds/foo/bar/api/.git/
 Created fresh repository.
 From https://gitlab.com/foo/bar/api
  * [new ref]         refs/pipelines/141504246 -> refs/pipelines/141504246
  * [new branch]      master                   -> origin/master
 Checking out 823e4b7a as master...
 Skipping Git submodules setup
Restoring cache
00:01
Downloading artifacts
Running before_script and script
00:01
 $ apk add --no-cache curl jq python py-pip
 fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
 fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
 ERROR: unsatisfiable constraints:
   python (missing):
     required by: world[python]
Running after_script
00:01
Uploading artifacts for failed job
00:02
 ERROR: Job failed: exit code 1

This same CI script worked on the same code a week ago. Judging by the logs, probably GitLab runner is unable to download/install python properly. Are there any alternative commands to install python and pip in GitLab-CI ? Or I should wait till GitLab fixes the bug?

3 Answers 3

6

This is not a GitLab issue. docker:stable removed the python package. Using docker image docker:19.03.8 fixed the problem.

Updated .gitlab-ci.yml

image: docker:19.03.8

services:
  - docker:dind

stages:
  - build

variables:
  REPOSITORY_URL: 708604513256.dkr.ecr.ap-south-1.amazonaws.com/foo-bar/api:latest

before_script:
  - apk add --no-cache curl jq python py-pip
  - pip install awscli

build:
  stage: build
  only:
    - master
  script:
    - $(aws ecr get-login --no-include-email --region ap-south-1)
    - docker build -t $REPOSITORY_URL .
    - docker push $REPOSITORY_URL

note: Although I am not sure if we can install awscli with docker:latest any other way. If that's possible, I think that would be a better solution.

1
  • 1
    There are other solutions available - pin your docker images to docker:19.03.10 or apk add python2 if you have to use docker:19.03.11 and later. See my answer (stackoverflow.com/a/62284538/1264251) for more information.
    – zorlem
    Jun 9, 2020 at 14:08
1

This is an unexpected side-effect after docker-library (https://hub.docker.com/_/docker/, https://github.com/docker-library/) developers updated the base image for their latest docker images from Alpine 3.11 to Alpine 3.12 (https://github.com/docker-library/docker/issues/240).

Alpine 3.12 removed the un-versioned python symlink and package alias (https://git.alpinelinux.org/aports/commit/?h=3.12-stable&id=5ad0ec7da1064361cc74d56edf7524960f49ef9b) and we have to use either python2 or python3 to install Python, or pin to a specific docker image version (which is almost always a good idea, instead of using :latest or :stable tags). docker:19.03.10 has python, while docker:19.03.11 and later don't have python, but python2 and python3.

1

@bopsi just FYI the format is invalid in the actual gitlab ci example you provided. It's correct in your actual response. In the gitlab CI example, it should be:

image: docker:19.03.8

cheers!

2
  • 2
    This should have been posted as a comment (or an edit, but you don't have enough reputation to edit) to the original answer you were addressing, not a separate answer.
    – zorlem
    Jun 9, 2020 at 18:18
  • Don't have enough rep to post a comment either :)
    – bwl1289
    Jun 10, 2020 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.