23

GitHub's Google Cloud Build integration does not detect a cloudbuild.yaml or Dockerfile if it is not in the root of the repository.

When using a monorepo that contains multiple cloudbuild.yamls, how can GitHub's Google Cloud Build integration be configured to detect the correct cloudbuild.yaml?

File paths:

services/api/cloudbuild.yaml
services/nginx/cloudbuild.yaml
services/websocket/cloudbuild.yaml

Cloud Build integration output:

Failed build

3 Answers 3

41

You can do this by adding a cloudbuild.yaml in the root of your repository with a single gcr.io/cloud-builders/gcloud step. This step should:

  1. Traverse each subdirectory or use find to locate additional cloudbuild.yaml files.
  2. For each found cloudbuild.yaml, fork and submit a build by running gcloud builds submit.
  3. Wait for all the forked gcloud commands to complete.

There's a good example of one way to do this in the root cloudbuild.yaml within the GoogleCloudPlatform/cloud-builders-community repo.

If we strip out the non-essential parts, basically you have something like this:

steps:
- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    for d in */; do
      config="${d}cloudbuild.yaml"
      if [[ ! -f "${config}" ]]; then
        continue
      fi

      echo "Building $d ... "
      (
        gcloud builds submit $d --config=${config}
      ) &
    done
    wait
5
  • 2
    Credit goes to @philippe-modard for this solution by the way. Oct 19, 2018 at 14:37
  • 3
    Thank you for your solution Rohan, so glad to have an answer! It's a shame this isn't supported natively, hopefully this thread will cause the cloudbuild developers to consider supporting this use case.
    – sabrehagen
    Oct 20, 2018 at 0:45
  • How would you do it if, like us, we are using a monorepo but only have 1 cloudbuild.yaml file located in /website/web/src - our triggger is set to that target, but when we run, it doesn't seem to be able to be located in that folder (for example, it can't find package.json)
    – kingloui
    Nov 13, 2019 at 18:04
  • 1
    Wouldn't this build every project in the repo every time a commit is made to any project, or is it smart enough to recognize if the commit didn't apply to that subdirectory?
    – swelljoe
    Apr 8, 2021 at 7:42
  • It builds everything. Apr 8, 2021 at 20:08
7

We are migrating to a mono-repo right now, and I haven't found any CI/CD solution that handles this well.

The key is to not only detect changes, but also any services that depend on that change. Here is what we are doing:

  • Requiring every service to have a MAKEFILE with a build command.
  • Putting a cloudbuild.yaml at the root of the mono repo
  • We then run a custom build step with this little tool (old but still seems to work) https://github.com/jharlap/affected which lists out all packages have changed and all packages that depend on those packages, etc.
  • then the shell script will run make build on any service that is affected by the change.

So far it is working well, but I totally understand if this doesn't fit your workflow.

Another option many people use is Bazel. Not the most simple tool, but especially great if you have many different languages or build processes across your mono repo.

2
  • Honestly, I think the way you're doing it is the right way. The other way (having multiple triggers and filters defined in Google Cloud Build) is less portable. I prefer having all the logic of which builds should be triggered within the monorepo itself. Oct 19, 2018 at 14:26
  • Yes, if you use a monorepo, you should also use a monorepo-compatible build tool, like Bazel. The intention with a monorepo is that a change can affect multiple apps, so it needs to know the dependency-relation, as you write - only tools like Bazel does this.
    – Jonas
    Aug 23, 2020 at 9:20
-2

You can create a build trigger for your repository. When setting up a trigger with cloudbuild.yaml for build configuration, you need to provide the path to the cloudbuild.yaml within the repository.

1
  • 2
    The point of the GitHub plugin is to auto-detect cloudbuild.yaml file so the manual creation of build triggers is not necessary.
    – sabrehagen
    Sep 15, 2018 at 2:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.