148

I recently hooked up my project with github actions for continuous integration. I created two separate jobs: the first one checks if the code in the pull request is accepted by our linter, and the second one checks if the code passes the test suite. I like that having two jobs like this shows up as two separate checkmarks in the Github webpage for the pull request:

enter image description here

The problem I'm having now is that there is some duplicated code in workflow YAML file: the first 3 steps, which install Lua and Luarocks. Not only is it annoying to maintain, but it also wastes CI minutes by running the same actions twice. Is there a way to avoid this? So that the setup code is only written in one place, and only runs once when the workflow executes?

But I am confused what would be the proper way to proceed:

  1. Should I create my own Github Action with the shared setup code?
  2. Should I create a Docker image that already has Lua and Luarocks pre-installed?
  3. Should I use a single job? Can I still have independent checkmarks for the linter and the test suite if they are steps of the same job?
  4. Something else?

Here is the current YAML file for my workflow:

name: Github Actions CI

on: [ pull_request ]

jobs:
    lint:
        name: Lint
        runs-on: ubuntu-latest
        steps:
            - uses: actions/checkout@v2
            - uses: leafo/[email protected]
            - uses: leafo/[email protected]

            - run: luarocks install luacheck
            - run: ./run-linter.sh

    test:
        name: Test
        runs-on: ubuntu-latest
        steps:
            - uses: actions/checkout@v2
            - uses: leafo/[email protected]
            - uses: leafo/[email protected]

            - run: luarocks install busted
            - run: ./build-project.sh
            - run: ./run-test-suite.sh

I tried searching for similar questions but couldn't find anything that exactly answered my question:

  1. Caching APT packages in GitHub Actions workflow: I can't use this solution because I don't have a way to precisely specify all the versions of all the dependencies that I am using, so that they may be cached. I also don't mind if separate runs of the workflow are not cached. I'm more worried about the code duplication.
  2. Github actions share workspace/artifacts between jobs? I don't want to have to manage uploading uploading artifacts to a separate service and then deleting them afterwards.
  3. Reuse portion of github action across jobs: In that question the only difference between the jobs is a single variable, so accepted answer is to use a build matrix. But I don't think a build matrix would work as well in my case, where only the setup code is the same?
1

4 Answers 4

51

As of today (August 2021) composite action is no longer limited to run. GitHub Actions: Reduce duplication with action composition

name: "Publish to Docker"
description: "Pushes built artifacts to Docker"

inputs:
  registry_username:
    description: “Username for image registry”
    required: true
  registry_password:
    description: “Password for image registry”
    required: true

runs:
  using: "composite"
  steps:
      - uses: docker/setup-buildx-action@v1

      - uses: docker/login-action@v1
        with:
          username: ${{inputs.registry_username}}
          password: ${{inputs.registry_password}}

      - uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          tags: user/app:latest

Old Answer

What you are looking for is composite action which help you reuse once defined set of steps.

jobs:
    lint:
        name: Lint
        runs-on: ubuntu-latest
        steps:
            - uses: octocat/say-hello@v1

            - run: luarocks install luacheck
            - run: ./run-linter.sh

    test:
        name: Test
        runs-on: ubuntu-latest
        steps:
            - uses: octocat/say-hello@v1

            - run: luarocks install busted
            - run: ./build-project.sh
            - run: ./run-test-suite.sh

octocat/say-hello/action.yml:

runs:
  using: "composite"
  steps: 
    - run: echo "Nice to meet you!"
      shell: pwsh

For more details you can also check's ADR here.

And why you can't simply run it once for all jobs, because each job may run on a different machine.

A job is a set of steps that execute on the same runner. By default, a workflow with multiple jobs will run those jobs in parallel. You can also configure a workflow to run jobs sequentially. For example, a workflow can have two sequential jobs that build and test code, where the test job is dependent on the status of the build job. If the build job fails, the test job will not run.

3
  • @AJNinja this is now supported! Commented Aug 26, 2021 at 15:55
  • Can you define this composite action in the same repo that will use it? For example I have 4 jobs that run when a PR is opened in a repo, but they all need the same series of "setup" steps. All examples I see are where the action lives in a separate repo Commented Jan 10, 2023 at 23:16
  • 1
    Yes. Of course, you can have composite action in the same repo. Take a look here: github.com/kmadof/github-actions-manual/blob/main/.github/… Commented Jan 11, 2023 at 11:20
27

There are 3 main approaches for code reusing in GitHub Actions:

There is an article describing their pros and cons.

In your case if the duplicated steps are in the single workflow you also can:

  1. extract them to the "preparation" job
  2. upload build artifacts
  3. add "preparation" job to "needs" key of both jobs
  4. download build artifact in both jobs
4
  • 2
    I tried using the upload artifacts route but it is horrendously slow. Granted I was also uploading the repo I checkout out with node_modules included. But, I don't think I should have to run checkout and npm i for each job that needs it. That's unnecessary duplication. Commented Jan 10, 2023 at 23:19
  • 1. You can cache node dependencies 2. You can extract repeated steps to the Composite Action that runs on the same worker in the context of your job, sharing the file system
    – Cardinal
    Commented Jan 12, 2023 at 16:40
  • 9
    Yes, but composite actions just allow you to group common tasks. Each job still needs to do those common tasks which is redundant. There should be a way to have 1 job that all other jobs rely on through the needs keyword that does what should be 1 time setup work Commented Jan 12, 2023 at 16:59
  • @TemporaryFix is absolutely right. There should be a way to run 1 setup job once in a machine, and have all other jobs be able to "need" on it, ensuring the setup is run only once, without uploading any artifact or having to cache anything. By sharing the same filesystem it would be enough. We can already do it with single-job runs, but they are ugly and lose the ability to parallelise jobs. It's just bad design on GitHub's part IMHO. Commented May 16 at 14:55
1

You can conditionally skip steps using matrix properties. It's an example from my action:

- name: Install dependencies on Debian
  if: contains(matrix.image, 'ubuntu') || contains(matrix.image, 'debian')
  run: |
    apt-get update -y
    apt-get install -y curl make ${{ matrix.cpp_compiler }}

As for caching, you need to use the hash of key files (such as the list of the installed packages) as the key of the cache. But Lua is extremely lightweight, so I guess that the installation of these extra packages takes only a few seconds.

-4

The below code snippet should do the work for you. Instead of creating two jobs, one for lint and another for test, you can use one job and create multiple tasks

name: Github Actions CI

on: [ pull_request ]

jobs:
    lint:
        name: Lint and Test
        runs-on: ubuntu-latest
        steps:
            - uses: actions/checkout@v2
            - uses: leafo/[email protected]
            - uses: leafo/[email protected]

            - name: Install Luacheck
              run: luarocks install luacheck

            - name: Run Lint Check
            - run: ./run-linter.sh

            - name: Install Busted
              run: luarocks install busted

            - name: Build Project
              run: ./build-project.sh

            - name: Run Test Suite
            - run: ./run-test-suite.sh

1
  • 2
    This doesn't allow the linter and unit tests to run in parallel, and it doesn't surface unit test failures until after lint failures are resolved. Commented Aug 24, 2021 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.