248

If a GitLab project is configured on GitLab CI, is there a way to run the build locally?

I don't want to turn my laptop into a build "runner", I just want to take advantage of Docker and .gitlab-ci.yml to run tests locally (i.e. it's all pre-configured). Another advantage of that is that I'm sure that I'm using the same environment locally and on CI.

Here is an example of how to run Travis builds locally using Docker, I'm looking for something similar with GitLab.

1

11 Answers 11

230

Since a few months ago this is possible using gitlab-runner:

gitlab-runner exec docker my-job-name

Note that you need both docker and gitlab-runner installed on your computer to get this working.

You also need the image key defined in your .gitlab-ci.yml file. Otherwise won't work.

Here's the line I currently use for testing locally using gitlab-runner:

gitlab-runner exec docker test --docker-volumes "/home/elboletaire/.ssh/id_rsa:/root/.ssh/id_rsa:ro"

Note: You can avoid adding a --docker-volumes with your key setting it by default in /etc/gitlab-runner/config.toml. See the official documentation for more details. Also, use gitlab-runner exec docker --help to see all docker-based runner options (like variables, volumes, networks, etc.).

Due to the confusion in the comments, I paste here the gitlab-runner --help result, so you can see that gitlab-runner can make builds locally:

   gitlab-runner --help
NAME:
   gitlab-runner - a GitLab Runner

USAGE:
   gitlab-runner [global options] command [command options] [arguments...]
   
VERSION:
   1.1.0~beta.135.g24365ee (24365ee)
   
AUTHOR(S):
   Kamil Trzciński <[email protected]> 
   
COMMANDS:
   exec         execute a build locally
   [...]
   
GLOBAL OPTIONS:
   --debug          debug mode [$DEBUG]
   [...]

As you can see, the exec command is to execute a build locally.

Even though there was an issue to deprecate the current gitlab-runner exec behavior, it ended up being reconsidered and a new version with greater features will replace the current exec functionality.

Note that this process is to use your own machine to run the tests using docker containers. This is not to define custom runners. To do so, just go to your repo's CI/CD settings and read the documentation there. If you wanna ensure your runner is executed instead of one from gitlab.com, add a custom and unique tag to your runner, ensure it only runs tagged jobs and tag all the jobs you want your runner to be responsible of.

24
  • 2
    @elboletaire How about running the whole build, not a single stage?
    – Nowaker
    Commented Aug 30, 2016 at 3:55
  • 18
    gitlab-runner exec is being deprecated after GitLab 10.0, vote gitlab.com/gitlab-org/gitlab-runner/issues/2797 to support a replacement before this happens
    – Alfageme
    Commented Oct 24, 2017 at 15:58
  • 2
    Thanks for pointing it out @Alfageme, apparently they're not going to deprecate it until a replacement is developed, see gitlab.com/gitlab-org/gitlab-runner/issues/2797#note_42944825 Commented Oct 24, 2017 at 16:39
  • 3
    @KongJinJie, you can do gitlab-runner exec docker --help and you'll see all the options. In your case, for env vars, is as simple as adding --env VARIABLE=value as a param of gitlab-runner Commented Jan 3, 2019 at 12:47
  • 4
    An FYI for anyone hoping to use this to test things locally - gitlab-runner exec does not support a whole swathe of things in Gitlab's CI config, perhaps most notably the extends key. Many others are not supported but also not listed on that page! So if you've used these features to make your CI config more maintainable or composable, gitlab-runner is not going to help.
    – detly
    Commented Jan 14, 2023 at 9:06
181

Deprecated since 2024-05 (see edit)

Only for gitlab-runner < 17.0.

I use this docker-based approach:

Edit: 2024-05

docker run --entrypoint bash --rm -w $PWD -v $PWD:$PWD -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:v15.11.1 -c 'git config --global --add safe.directory "*";gitlab-runner exec docker test'

Thanks to user Ivo Smits who reported a successful test with gitlab-runner:v15.11.1

Edit: 2022-10

For all git versions > 2.35.2. You must add safe.directory within the container to avoid fatal: detected dubious ownership in repository at.... This also true for patched git versions < 2.35.2. The old command will not work anymore.

Details

0. Create a git repo to test this answer

mkdir my-git-project
cd my-git-project
git init
git commit --allow-empty -m"Initialize repo to showcase gitlab-runner locally."

1. Go to your git directory

cd my-git-project

2. Create a .gitlab-ci.yml

Example .gitlab-ci.yml

image: alpine

test:
  script:
    - echo "Hello Gitlab-Runner"

3. Create a docker container with your project dir mounted

docker run -d \
  --name gitlab-runner \
  --restart always \
  -v $PWD:$PWD \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

(-d) run container in background and print container ID

(--restart always) or not?

(-v $PWD:$PWD) Mount current directory into the current directory of the container - Note: On Windows you could bind your dir to a fixed location, e.g. -v ${PWD}:/opt/myapp. Also $PWD will only work at powershell not at cmd

(-v /var/run/docker.sock:/var/run/docker.sock) This gives the container access to the docker socket of the host so it can start "sibling containers" (e.g. Alpine).

(gitlab/gitlab-runner:latest) Just the latest available image from dockerhub.

4. Execute with

Avoid fatal: detected dubious ownership in repository at... More info

docker exec -it -w $PWD gitlab-runner git config --global --add safe.directory "*"

Actual execution

docker exec -it -w $PWD gitlab-runner gitlab-runner exec docker test
#                ^          ^           ^            ^     ^      ^
#                |          |           |            |     |      |
#               (a)        (b)         (c)          (d)   (e)    (f)

(a) Working dir within the container. Note: On Windows you could use a fixed location, e.g. /opt/myapp.

(b) Name of the docker container

(c) Execute the command "gitlab-runner" within the docker container

(d)(e)(f) run gitlab-runner with "docker executer" and run a job named "test"

5. Prints

...
Executing "step_script" stage of the job script
$ echo "Hello Gitlab-Runner"
Hello Gitlab-Runner
Job succeeded
...

Note: The runner will only work on the commited state of your code base. Uncommited changes will be ignored. Exception: The .gitlab-ci.yml itself does not have be commited to be taken into account.

Note: There are some limitations running locally. Have a look at limitations of gitlab runner locally.

17
  • 20
    Very useful examples here help to understand how it works Commented Sep 4, 2021 at 14:03
  • 1
    Note: it is important to make sure you use -v $PWD:$PWD because of the docker in docker situation (otherwise you will get an error fatal: Could not read from remote repository.). See gitlab.com/gitlab-org/gitlab-runner/-/issues/4574
    – vab2048
    Commented Nov 30, 2021 at 22:20
  • 1
    Step 3 does not work in windows terminal. Use docker run -d --name gitlab-runner --restart always -v ${PWD}:/usr/src/app -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest instead. Only the mount file argument has been changed. $PWD has been replaced by ${PWD} to work in windows. Also you cannot use the same path in the docker container as it runs linux, so I replaced it with a generic path: /usr/src/app Commented Dec 12, 2021 at 16:20
  • 4
    For those who want to use a single docker run command: docker run --rm -w $PWD -v $PWD:$PWD -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest exec docker test
    – Jack
    Commented Dec 20, 2021 at 17:37
  • 3
    It appears that the exec method has been removed from the latest builds. Luckily docker allows to use an older version where it's still supported: docker run --entrypoint bash --rm -w $PWD -v $PWD:$PWD -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:v15.11.1 -c 'git config --global --add safe.directory "*";gitlab-runner exec docker test' Thanks for this solution!
    – Ivo Smits
    Commented May 30 at 16:31
30

I'm currently working on making a gitlab runner that works locally. Still in the early phases, but eventually it will become very relevant. It doesn't seem like gitlab want/have time to make this, so here you go. https://github.com/firecow/gitlab-runner-local

4
  • 1
    npm install gives an error but installing with Linux library works like a charm. Thanks! I can use this!
    – Maarten
    Commented Apr 28, 2021 at 16:34
  • *Linux Binary - Months later I have a new laptop and had to test pipelines locally again. I am glad I found this answer again :)
    – Maarten
    Commented Sep 25, 2021 at 10:20
  • Does this support services and reading a gitlab-runner toml config file?
    – Farhood ET
    Commented Aug 2, 2022 at 9:37
  • Cool project. Is this an effort to fully reverse engineer the Gitlab Runner, or is it a wrapper around the official [gitlab runner](brew install gitlab-ci-local)?
    – chadrik
    Commented Aug 14 at 3:06
7

The GitLab runner appears to not work on Windows yet and there is an open issue to resolve this.

So, in the meantime I am moving my script code out to a bash script, which I can easily map to a docker container running locally and execute.

In this case I want to build a docker container in my job, so I create a script 'build':

#!/bin/bash

docker build --pull -t myimage:myversion .

in my .gitlab-ci.yaml I execute the script:

image: docker:latest

services:
- docker:dind

before_script:
- apk add bash

build:
stage: build
script:
- chmod 755 build
- build

To run the script locally using powershell I can start the required image and map the volume with the source files:

$containerId = docker run --privileged -d -v ${PWD}:/src docker:dind

install bash if not present:

docker exec $containerId apk add bash

Set permissions on the bash script:

docker exec -it $containerId chmod 755 /src/build

Execute the script:

docker exec -it --workdir /src $containerId bash -c 'build'

Then stop the container:

docker stop $containerId

And finally clean up the container:

docker container rm $containerId
2
  • This requires a Dockerfile, which you don't mention.
    – Cerin
    Commented Jul 23, 2019 at 17:00
  • @Cerin what dockerfile is required? docker:dind is the official docker image, I did not create it. Commented Jul 29, 2019 at 16:31
5

If you are running Gitlab using the docker image there: https://hub.docker.com/r/gitlab/gitlab-ce, it's possible to run pipelines by exposing the local docker.sock with a volume option: -v /var/run/docker.sock:/var/run/docker.sock. Adding this option to the Gitlab container will allow your workers to access to the docker instance on the host.

2
  • I'm currently trying to execute a task in the .gitlab-ci.yml file in my project, on a Runner deployed as a Docker container. Do I need to bind mount the src code of my project into the Runner in order for it to find/run the task? Or is this somehow possible with what you said in your answer, i.e. connecting the to remote client like in Docker machine 'eval "$(docker-machine env default)"'?
    – Greg Brown
    Commented Sep 25, 2018 at 21:39
  • 1
    @GregBrown or if sb else was wondering, have a look a this issue, eg.: docker run --rm -v $PWD:$PWD -v /var/run/docker.sock:/var/run/docker.sock -v /opt/gitlab-runner/config:/etc/gitlab-runner --workdir $PWD gitlab/gitlab-runner exec docker deploy. Option --workdir is the key.
    – The_Ham
    Commented Jan 20, 2021 at 22:22
5

Another approach is to have a local build tool that is installed on your pc and your server at the same time. So basically, your .gitlab-ci.yml will basically call your preferred build tool.

Here an example .gitlab-ci.yml that i use with nuke.build:

stages:
    - build
    - test
    - pack

variables:
    TERM: "xterm" # Use Unix ASCII color codes on Nuke

before_script:
    - CHCP 65001  # Set correct code page to avoid charset issues

.job_template: &job_definition
  except:
    - tags

build:
    <<: *job_definition
    stage: build
    script:
        - "./build.ps1"

test:
    <<: *job_definition
    stage: test
    script:
        - "./build.ps1 test"
    variables:
        GIT_CHECKOUT: "false"

pack:
    <<: *job_definition
    stage: pack
    script:
        - "./build.ps1 pack"
    variables:
        GIT_CHECKOUT: "false"
    only:
        - master
    artifacts:
        paths:
            - output/

And in nuke.build i've defined 3 targets named like the 3 stages (build, test, pack)

In this way you have a reproducible setup (all other things are configured with your build tool) and you can test directly the different targets of your build tool.

(i can call .\build.ps1 , .\build.ps1 test and .\build.ps1 pack when i want)

4

I am on Windows using VSCode with WSL

I didn't want to register my work PC as a runner so instead I'm running my yaml stages locally to test them out before I upload them

$ sudo apt-get install gitlab-runner
$ gitlab-runner exec shell build

yaml

image: node:10.19.0 # https://hub.docker.com/_/node/
# image: node:latest

cache:
  # untracked: true
  key: project-name
  # key: ${CI_COMMIT_REF_SLUG} # per branch
  # key:
  #   files:
  #     - package-lock.json # only update cache when this file changes (not working) @jkr
  paths:
    - .npm/
    - node_modules
    - build

stages:
  - prepare # prepares builds, makes build needed for testing
  - test # uses test:build specifically @jkr
  - build
  - deploy

# before_install:

before_script:
  - npm ci --cache .npm --prefer-offline

prepare:
  stage: prepare
  needs: []
  script:
    - npm install

test:
  stage: test
  needs: [prepare]
  except:
    - schedules
  tags:
    - linux
  script:
    - npm run build:dev
    - npm run test:cicd-deps
    - npm run test:cicd # runs puppeteer tests @jkr
  artifacts:
    reports:
      junit: junit.xml
    paths:
      - coverage/

build-staging:
  stage: build
  needs: [prepare]
  only:
    - schedules
  before_script:
    - apt-get update && apt-get install -y zip
  script:
    - npm run build:stage
    - zip -r build.zip build
  # cache:
  #   paths:
  #     - build
  #   <<: *global_cache
  #   policy: push
  artifacts:
    paths:
      - build.zip

deploy-dev:
  stage: deploy
  needs: [build-staging]
  tags: [linux]
  only:
    - schedules
  #   # - branches@gitlab-org/gitlab
  before_script:
    - apt-get update && apt-get install -y lftp
  script:
    # temporarily using 'verify-certificate no'
    # for more on verify-certificate @jkr: https://www.versatilewebsolutions.com/blog/2014/04/lftp-ftps-and-certificate-verification.html
    # variables do not work with 'single quotes' unless they are "'surrounded by doubles'"
    - lftp -e "set ssl:verify-certificate no; open mediajackagency.com; user $LFTP_USERNAME $LFTP_PASSWORD; mirror --reverse --verbose build/ /var/www/domains/dev/clients/client/project/build/; bye"
  # environment:
  #   name: staging
  #   url: http://dev.mediajackagency.com/clients/client/build
  #   # url: https://stg2.client.co
  when: manual
  allow_failure: true

build-production:
  stage: build
  needs: [prepare]
  only:
    - schedules
  before_script:
    - apt-get update && apt-get install -y zip
  script:
    - npm run build
    - zip -r build.zip build
  # cache:
  #   paths:
  #     - build
  #   <<: *global_cache
  #   policy: push
  artifacts:
    paths:
      - build.zip

deploy-client:
  stage: deploy
  needs: [build-production]
  tags: [linux]
  only:
    - schedules
    # - master
  before_script:
    - apt-get update && apt-get install -y lftp
  script:
    - sh deploy-prod
  environment:
    name: production
    url: http://www.client.co
  when: manual
  allow_failure: true
2
  • what about docker ? You have specified 'image' in your yaml Commented Sep 4, 2020 at 14:03
  • @ShubhamTakode I initially went that route but getting everything to work smoothly on WSL proved to be more effort that I was wanting to put into this issue.
    – Jacksonkr
    Commented Sep 6, 2020 at 20:39
4

I recommend using gitlab-ci-local https://github.com/firecow/gitlab-ci-local It's able to run specific jobs as well. It's a very cool project and I have used it to run simple pipelines on my laptop.

3

The idea is to keep check commands outside of .gitlab-ci.yml. I use Makefile to run something like make check and my .gitlab-ci.yml runs the same make commands that I use locally to check various things before committing.
This way you'll have one place with all/most of your commands (Makefile) and .gitlab-ci.yml will have only CI-related stuff.

2

I have written a tool to run all GitLab-CI job locally without have to commit or push, simply with the command ci-toolbox my_job_name.

The URL of the project : https://gitlab.com/mbedsys/citbx4gitlab

2
  • How mature is this tool? Apparently the last release was around 6 months ago. Can this be used in a production environment?
    – Farhood ET
    Commented Aug 2, 2022 at 9:39
  • You should try this tool yourself to do your own opinion... I use it every day, and it is used from more than 5 years in two companies by teams around 10 to 20 persons. Commented Aug 2, 2022 at 11:18
2

Years ago I build this simple solution with Makefile and docker-compose to run the gitlab runner in docker, you can use it to execute jobs locally as well and should work on all systems where docker works:

https://gitlab.com/1oglop1/gitlab-runner-docker

There are few things to change in the docker-compose.override.yaml

version: "3"
services:
    runner:
      working_dir: <your project dir>
      environment:
        - REGISTRATION_TOKEN=<token if you want to register>
      volumes:
        - "<your project dir>:<your project dir>"

Then inside your project you can execute it the same way as mentioned in other answers:

docker exec -it -w $PWD runner gitlab-runner exec <commands>..

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.