225

Trying to use Github's beta actions, I have two jobs, one that builds the code and then one that will deploy code. However, I can't seem to get the build artifact in deploy job.

My latest attempt is to manually set a container image with the same volumes for each job, according to docs this should be solution: https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idcontainervolumes

Sets an array of volumes for the container to use. You can use volumes to share data between services or other steps in a job. You can specify named Docker volumes, anonymous Docker volumes, or bind mounts on the host.

Workflow

name: CI
on:
  push:
    branches:
    - master
    paths:
    - .github/workflows/server.yml
    - server/*
jobs:
  build:
    runs-on: ubuntu-latest
    container:
      image: docker://node:10
      volumes:
      - /workspace:/github/workspace
    steps:
    - uses: actions/checkout@master
    - run: yarn install
      working-directory: server
    - run: yarn build
      working-directory: server
    - run: yarn test
      working-directory: server
    - run: ls
      working-directory: server
  deploy:
    needs: build
    runs-on: ubuntu-latest
    container:
      image: docker://google/cloud-sdk:latest
      volumes:
      - /workspace:/github/workspace
    steps:
      - uses: actions/checkout@master
      - run: ls
        working-directory: server
      - run: gcloud --version

The first job (build) has a build directory, but when the second job (deploy) runs it doesn't and only contains the source code.

This project is a mono repo with code I'm trying to deploy being under path server hence all the working-directory flags.

6
  • 5
    See stackoverflow.com/questions/57509118/… - the Workflow syntax docs say "Each job runs in a fresh instance of the virtual environment specified by runs-on." My guess (I'm not in the beta so I'm just guessing) is that your deploy job would either need to become a step in the build job, or would need to reproduce the build steps again in the new container. (Minus, perhaps, the yarn test step since you already know that it succeeded).
    – rmunn
    Commented Aug 19, 2019 at 7:42
  • 2
    Did you ever find an answer for this? I'm trying to figure out how to do this as well.. From what i've read jobs are supposed to share the workspace filesystem, but it doesn't seem to be the case.
    – Joseph
    Commented Aug 20, 2019 at 17:56
  • 1
    @Joseph nope, I'm just running one job and using custom docker image. I believe issue sits with GitHub and is likely due to the transitioning from HCL to YML syntax. Odd that they plan to drop HCL at the end of September and the basic ability to share artefacts between jobs doesn't work yet. Hopefully, in a months time, it will be resolved. Commented Aug 21, 2019 at 8:29
  • "You can use volumes to share data between services or other steps in a job." That means sharing data within a single job between steps. It's not for sharing data between steps or workflows. Commented Jan 31, 2020 at 15:43
  • 4
    3 years, 5 months later, and this is still not possible without using artifacts or cache. GitHub, if you will ever read this, a simple keyword after: <job_id> would be amazing. Commented Jan 18, 2023 at 23:39

6 Answers 6

198

You can use the Github Actions upload-artifact and download-artifact to share data between jobs. For example:

Job 1:

    steps:
      - uses: actions/checkout@v4
      - run: mkdir -p path/to/artifact
      - run: echo hello > path/to/artifact/world.txt
      - uses: actions/upload-artifact@v4
        with:
          name: my-artifact
          path: path/to/artifact

Job 2:

    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          name: my-artifact
          path: path/to/artifact
      - run: cat path/to/artifact/world.txt
8
  • 3
    you can now also delete artifacts: github.com/marketplace/actions/delete-run-artifacts github.com/marketplace/actions/purge-artifacts
    – Chris
    Commented Mar 11, 2020 at 18:54
  • 6
    while this is indeed an option, for anything serious (imagine 50 docker images shared), using this approach takes tens of minutes. Since github does not offer a proper solution, we switched to circleci, which cut the builds by at least 1 hour. its very, very weird not to have such an option
    – Eugene
    Commented Jul 31, 2022 at 6:44
  • 2
    The order of the steps in 2nd job is very important. checkout action deletes everything :) Commented May 17, 2023 at 11:01
  • 1
    Is this also needed between steps? if I have a JAR file that is build in step 1, do I need to upload/download it to be used in step 4?
    – Rafs
    Commented Jun 12 at 15:16
  • 1
    No @Rafs, steps within a single job are all "incremental" you could say. Any file made and left by one step is available to the next. Commented Jun 26 at 21:43
51

For those interested in sharing a Docker image between two jobs, here is how I did it:

jobs:
  docker-build:
    name: Docker build
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Build Docker image
        run: |
          docker build -t foo/bar:$GITHUB_SHA
          mkdir -p path/to/artifacts
          docker save foo/bar:$GITHUB_SHA > path/to/artifacts/docker-image.tar
          
      - name: Temporarily save Docker image
        uses: actions/upload-artifact@v2
        with:
          name: docker-artifact
          path: path/to/artifacts
          retention-days: 1

  docker-deploy:
    name: Deploy to Docker Hub
    runs-on: ubuntu-latest
    needs: docker-build
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Retrieve saved Docker image
        uses: actions/download-artifact@v2
        with:
          name: docker-artifact
          path: path/to/artifacts

      - name: Docker load
        run: |
          cd path/to/artifacts
          docker load < docker-image.tar
          # docker_build_push.sh

Very inspired by https://github.com/unfor19/install-aws-cli-action/actions/runs/400601222/workflow

Merci @unfor19

4
  • 1
    For anyone coming across this and wondering how to avoid cluttering up your artifacts since github doesn't provide an action for that: there are several community actions, e.g. github.com/marketplace/actions/delete-artifact
    – Alan Plum
    Commented Jan 25, 2021 at 18:38
  • 4
    @AlanPlum Well said. We do clean up artifacts now with the parameter retention-days. I'll update my answer. Thanks.
    – Thomas
    Commented Jan 26, 2021 at 19:24
  • 1
    this is indeed an option, but if you have many images, this is a very painful and slow option. Other CI/CD frameworks have this by default, github does not.
    – Eugene
    Commented Jul 31, 2022 at 6:46
  • 1
    retention-days is the key! Otherwise Github gonna waste your money
    – orkenstein
    Commented Aug 24, 2022 at 20:27
31

Use Cache or Artifacts Upload/Download

Caching is used to re-use data/files between jobs or workflows while Artifacts are used to save files after workflow ended.

30

GitHub Actions – Artifacts v4 is now Generally Available (Dec. 2023)

We have released new versions (v4) of actions/upload-artifact and actions/download-artifact.
Customers are encouraged to update their workflows to use the v4 artifact actions at their earliest convenience.

This version of the artifact actions include performance improvements and new features, however there are also key differences from previous versions that may require updates to your workflows.

  • Artifacts will be scoped to a job rather than a workflow.
    This allows the artifact to become immediately available to download from the API after being uploaded.
  • Multiple jobs in the same run cannot upload to the same artifact.
  • A single job can upload a maximum of 10 artifacts.
  • Artifacts v4 is not cross-compatible with previous versions.
    For example, an artifact uploaded using v3 cannot be used with actions/download-artifact@v4.

This version of artifacts is only available to GitHub.com customers today but we will be extending support to GitHub Enterprise Server (GHES) customers in the future.

A blog post will be published in the new year with more details around why we made these improvements and what's coming next.
To learn more about what is included in v4, visit actions/upload-artifact and actions/download-artifact repositories.


If you are using the upload/download GitHub Actions, beware of the structure of the artifact.

Starting January 2020, see "GitHub Actions: Changes to artifact download experience":

We have changed the artifact download experience in GitHub Actions so it no longer adds an extra root directory to the downloaded archive.

Previously, if you uploaded the following files and folders as an artifact named foo, the downloaded archive would contain the following structure:

foo/
|-- file1.txt
|-- dir1/
     |-- dir1-file1.txt

Now, you will get an archive that only contains the files and folders you uploaded:

file1.txt
  dir1/
  |-- dir1-file1.txt
1

Each job runs on the separate runner.

Upload artifact action implementation

Github Action "actions/upload-artifact@v3" uploads the files from provided path to storage container location.

In next job when you run action "actions/download-artifact@v3" , it downloads the artifact from 'storage container location' where previous job uploaded the artifacts to provided path.

Implementation for download artifact and displaying download path

For more information refer below links,

Github action for upload artifacts

Github action for download artifacts

0

Based on the answer on using caches and the answer for docker images, here is how I got it to work:

  get-docker-image:
    steps:
      - name: Create cache for docker image
        uses: actions/cache@v4
        with:
          path: images
          key: image-key

      - name: Download docker image from source
        run: |
          mkdir -p images
          cd images
          docker login registry
          docker pull path/image:tag
          docker save path/image:tag | gzip > docker-image.tgz

  user-docker-image:
    needs:
      - get-docker-image
    steps:
      - name: Restore cached docker image
        uses: actions/cache/restore@v4
        with:
          path: images
          key: image-key

      - name: Use the image
        working-directory: images
        run: |
          docker load < docker-image.tgz
          // do something with the image

Note that using caches is expected to be somewhat faster than upload/download.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.