8

In my project we are using nodejs with typescript for google cloud app engine app development. We have our own build mechanism to compile ts files into javascript ,then collect them into a complete runable package, so that we don't want to relay on google cloud to install dependencies, instead we want to upload all node packages inside the node_modules to google cloud.

But it seems google cloud will always ignore the node_modules folder and run npm install during the deployment. Even I tried to remove 'skip_files: - ^node_modules$' from app.yaml, it doesn't work, google cloud will always install packages by itself.

Does anyone have ideas of this of deploy node app with node_modules together? Thank you.

3 Answers 3

5

I observed the same issue.

My workaround was to rename node_modules/ to node_modules_hack/ before deploying. This prevents AppEngine from removing it.

I restore it to the original name on installation, with the following (partial) package.json file:

  "__comments": [
    "TODO: Remove node_modules_hack once AppEngine stops stripping node_modules/"
  ],
  "scripts": {
    "install": "mv -fn node_modules_hack node_modules",
    "start": "node server.js"
  },

You can confirm that AppEngine strips your node_modules/ by looking at the Docker image it generates. You can find it on the Images page. They give you a commandline that you can run on the cloud console to fetch it. Then you can run docker run <image_name> ls to see your directory structure. The image is created after npm install, so once you use the workaround above, you'll see your node_modules/ there.

2
  • mv -fn node_modules_hack node_modules <-- what language is that?
    – codeMonkey
    Jan 31, 2018 at 1:23
  • 1
    I am not sure if this is a good idea because there are a lot of npm packages that are platform (os) specific and pre installed node_modules might fail without you having a clue as to what happened
    – Yatin Gera
    Jul 11, 2018 at 6:57
1

The newest solution is to allow node_modules in .gcloudignore.

Below's the default .gcloudignore (one that initial execution of gcloud app deploy generates if you don't have one already) with the change you need:

# This file specifies files that are *not* uploaded to Google Cloud Platform
# using gcloud. It follows the same syntax as .gitignore, with the addition of
# "#!include" directives (which insert the entries of the given .gitignore-style
# file at that point).
#
# For more information, run:
#   $ gcloud topic gcloudignore
#
.gcloudignore
# If you would like to upload your .git directory, .gitignore file or files
# from your .gitignore file, remove the corresponding line
# below:
.git
.gitignore

# Node.js dependencies:
# node_modules/ # COMMENT OR REMOVE THIS LINE
1

Allowing node_modules in .gcloudignore no longer works.

App Engine deployment is switched to buildpacks since Oct/Nov 2020. Cloud Build step triggered by it will always remove uploaded node_modules folder and reinstall dependencies using yarn or npm.

Here is the related buildpack code: https://github.com/GoogleCloudPlatform/buildpacks/blob/89f4a6ba669437a47b482f4928f974d8b3ee666d/cmd/nodejs/yarn/main.go#L60

This is a desirable behaviour since uploaded node_modules could come from a different platform and could break compatibility with Linux runner used to run your app in App Engine environment.

So, in order to skip npm/yarn dependencies installation in Cloud Build, I would suggest to:

  • Use Linux runner CI with the same Node version you are using in the App Engine environment.

  • Create tar archive with your node_modules, to not upload multitude of files on each gcloud app deploy.

  • Keep node_modules dir ignored in .gcloudignore.

  • Unpack node_modules.tar.gz archive in preinstall script. Don't forget to keep backward compatibility in case tar archive is missing (local development, etc.):

{
  "scripts": {
    "preinstall": "test -f node_modules.tar.gz && tar -xzf node_modules.tar.gz && rm -f node_modules.tar.gz || true"
  }
}

Note ... || true thing. This will ensure preinstall script returns zero exit code no matter what, and yarn/npm install will continue.

Github Actions workflow to pack and upload your dependencies for App Engine deployment could look like this:

  deploy-gae:
    name: App Engine Deployment
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      # Preferable to use the same version as in GAE environment
      - name: Set Node.js version
        uses: actions/setup-node@v2
        with:
          node-version: '14.15.4'    

      - name: Save prod dependencies for GAE upload
        run: |
          yarn install --production=true --frozen-lockfile --non-interactive
          tar -czf node_modules.tar.gz node_modules
          ls -lah node_modules.tar.gz | awk '{print $5,$9}'

      - name: Deploy
        run: |
          gcloud --quiet app deploy app.yaml --no-promote --version "${GITHUB_ACTOR//[\[\]]/}-${GITHUB_SHA:0:7}"

This is just an expanded version of the initially suggested hack.

Note: In case you have a gcp-build script in your package.json you will need to create two archives (one for production dependencies and one for dev) and modify preinstall script to unpack the one currently needed (depending on the NODE_ENV set by buildpack).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.