35

I'm making an app using Google Calendar API, and planning to build it on Heroku.
I have a problem about authentication. Usually I use credential json file for that, but this time I don't want to upload it on Heroku for security reason.
How can I make authentiation on Heroku?

For now, I put my json to an env variable, and use oauth2client's from_json method.

def get_credentials():
    credentials_json = os.environ['GOOGLE_APPLICATION_CREDENTIALS']
    credentials = GoogleCredentials.from_json(credentials_json)
    if not credentials or credentials.invalid:
        flow = client.flow_from_clientsecrets(CLIENT_SECRET_FILE, SCOPES)
        flow.user_agent = APPLICATION_NAME
        if flags:
            credentials = tools.run_flow(flow, store, flags)
        else: # Needed only for compatibility with Python 2.6
            credentials = tools.run(flow, store)
        print('Storing credentials to ' + credential_path)
    return credentials 

But this code isn't perfect. If the credentials is invalid, I want the code to write the new credentials to the env variable, not to a new file.
Is there any better way?

14 Answers 14

41

I spent an entire day to find the solution because it's tricky. No matter your language, the solution will be the same.

1 - Declare your env variables from in Heroku dashboard like :

The GOOGLE_CREDENTIALS variable is the content of service account credential JSON file as is. The GOOGLE_APPLICATION_CREDENTIALS env variable in the string "google-credentials.json"

2 - Once variables are declared, add the builpack from command line :

$ heroku buildpacks:add https://github.com/elishaterada/heroku-google-application-credentials-buildpack

3 - Make a push. Update a tiny thing and push.

4 - The buildpack will automatically generate a google-credentials.json and fill it with the content of the google credentials content.

If you failed at something, it will not work. Check the content of the google-credentials.json with the Heroku bash.

7
  • 2
    FYI this buildpack currently does not work, you will probably have to use the fork by "gerywahyunugraha" of that buildpack. Dec 8, 2019 at 13:17
  • Directory cannot create or find, any idea why?
    – A. Amini
    Mar 22, 2020 at 14:57
  • I tried this answer but I got an error message, can someone help me? : Error: You have to specify credentials key file for Google Cloud Storage to work.
    – Rooney
    Apr 14, 2020 at 17:36
  • 5
    https://github.com/gerywahyunugraha/heroku-google-application-credentials-buildpack use this fork which resolves some bugs Jan 13, 2021 at 17:16
  • 1
    Can someone also add the code example that uses this solution?
    – Matej J
    Feb 17, 2021 at 14:55
15

The buildpack mentioned by Maxime Boué is not working anymore because of the Heroku updates(18+). However, below is a similar buildpack which is working. It is actually a fork from the previous one.

1
  • It now shows ; 2023-05-05T22:57:48.614081+00:00 app[web.1]: bash: line 1: gunicorn: command not found May 5, 2023 at 23:02
14

The recommended buildpack doesn't work anymore. Here's a quick, direct way to do the same:

  1. Set config variables:
heroku config:set GOOGLE_APPLICATION_CREDENTIALS=gcp_key.json
heroku config:set GOOGLE_CREDENTIALS=<CONTENTS OF YOU GCP KEY>

The GOOGLE_CREDENTIALS is easier to set in the Heroku dashboard.

  1. Create a .profile file in your repo with a line to write the json file:
echo ${GOOGLE_CREDENTIALS} > /gcp_key.json

.profile is run every time the container starts.

  1. Obviously, commit the changes to .profile and push to Heroku, which will trigger a rebuild and thus create that gcp_key.json file.

5
  • It worked for me! Thank you so much! I've used recommended buildpack but it only misconfigured my nodejs buildpack deployment, didn't know why. Aug 11, 2021 at 5:59
  • That's the best answer, worked perfectly! Thanks, man
    – Leonardo
    Dec 27, 2021 at 21:34
  • Why is the credentials file created at /app instead of in the root? Jan 5, 2022 at 1:29
  • I get "cannot create /app/gcp_key.json: Directory nonexistent"
    – Teddy Ward
    Dec 6, 2022 at 5:49
  • I haven't used Heroku in a while, but I've got a feeling they changed the directory they put our apps in. Which means echo ${GOOGLE_CREDENTIALS} > /app/gcp_key.json should be with echo ${GOOGLE_CREDENTIALS} > /gcp_key.json
    – Chrisjan
    Jan 13, 2023 at 15:40
12

If anyone is still looking for this, I've just managed to get this working for Google Cloud Storage by storing the JSON directly in an env variable (no extra buildpacks).

You'll need to place the json credentials data into your env vars and install google-auth

Then, parse the json and pass google credentials to the storage client:

from google.cloud import storage
from google.oauth2 import service_account

# the json credentials stored as env variable
json_str = os.environ.get('GOOGLE_APPLICATION_CREDENTIALS')
# project name
gcp_project = os.environ.get('GCP_PROJECT') 

# generate json - if there are errors here remove newlines in .env
json_data = json.loads(json_str)
# the private_key needs to replace \n parsed as string literal with escaped newlines
json_data['private_key'] = json_data['private_key'].replace('\\n', '\n')

# use service_account to generate credentials object
credentials = service_account.Credentials.from_service_account_info(
    json_data)

# pass credentials AND project name to new client object (did not work wihout project name)
storage_client = storage.Client(
    project=gcp_project, credentials=credentials)

Hope this helps!

EDIT: Clarified that this was for Google Cloud Storage. These classes will differ for other services, but from the looks of other docs the different Google Client classes should allow the passing of credentials objects.

4
  • So you just copy paste the whole json into a config var?
    – strivn
    Jun 24, 2020 at 8:43
  • @IvanWiryadi Yep! Just copy and paste the json object to heroku env config. If you're testing locally with an .env file make sure to delete any new lines.
    – Tom
    Jun 25, 2020 at 9:05
  • this helped me. Oct 9, 2020 at 21:59
  • Works great for Python. Thanks!
    – Julius
    Mar 15, 2022 at 8:32
4

A more official Heroku documentation in this topic: https://elements.heroku.com/buildpacks/buyersight/heroku-google-application-credentials-buildpack

I also used buyersight's buildpack and it was the only one, which worked for me

0
4

As state above, the official Heroku documentation works (https://elements.heroku.com/buildpacks/buyersight/heroku-google-application-credentials-buildpack).

For PHP Laravel users though, the config variable GOOGLE_APPLICATION_CREDENTIALS should be set to ../google-credentials.json. Otherwise, PHP will not find the file.

Screenshot from Heroku

1
2

I know this is old, but there is another alternative - ie to "split" the json file and store each important field as its own environment variable.

Something like:

PRIVATE_KEY_ID=qsd
PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n"
[email protected]
CLIENT_ID=7777
CLIENT_X509_CERT_URL=https://www.googleapis.com/robot/v1/metadata/x509/whatever.iam.gserviceaccount.com

This can be in a .env file locally, and put on heroku using the UI or heroku config:set commands

Then in the python file, you can initialize the ServiceAccount using a dict instead of a JSON

CREDENTIALS = {
    "type": "service_account",
    "project_id": "iospress",
    "private_key_id": os.environ["PRIVATE_KEY_ID"],
    "private_key": os.environ["PRIVATE_KEY"],
    "client_email": os.environ["CLIENT_EMAIL"],
    "client_id": os.environ["CLIENT_ID"],
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": os.environ["CLIENT_X509_CERT_URL"]
}

credentials = ServiceAccountCredentials.from_json_keyfile_dict(CREDENTIALS, SCOPES)

It's a bit more verbose than some of the options presented here, but it works without any buildback or other

1

In case you do not want to use buildpack

1 - Add env variables in Heroku via dashboard or CLI:

GOOGLE_CREDENTIALS variable is the content of the service account credential JSON file. GOOGLE_APPLICATION_CREDENTIALS = google-credentials.json

enter image description here

2 - Create a file called .profile on the root of your project with the following content

 echo ${GOOGLE_CREDENTIALS} > /app/google-credentials.json

3 - Push your code

4 - During startup, the container starts a bash shell that runs any code in $HOME/.profile before executing the dyno’s command.

Note: For Laravel projects GOOGLE_APPLICATION_CREDENTIALS = ../google-credentials.json

0

You can use the Heroku Platform API to update Heroku env vars from within your app.

0

It seems that those buildpacks where you can upload the credentials.json file are not working as expected. I finally managed with Lepinsk's buildpack (https://github.com/lepinsk/heroku-google-cloud-buildpack.git), which requires all keys and values to be set as config vars in Heroku. It does do the job though, so lots of thanks for that!

0

I have done this like below:

  • Created two env variable
    1. CREDENTIALS - base64 encoded value of google api credential
    2. CONFIG_FILE_PATH - /app/.gcp/key.json (this file we will create it at run time. in heroku preinstall phase as below)

Create a preinstall script.

"heroku-prebuild": "bash preinstall.sh",

And in preinstall.sh file, decode CREDENTIALS and create a config file and update it there.

if [ "$CREDENTIALS" != "" ]; then


echo "Detected credentials. Adding credentials" >&1
  echo "" >&1

  # Ensure we have an gcp folder
  if [ ! -d ./.gcp ]; then
    mkdir -p ./.gcp
    chmod 700 ./.gcp
  fi

  # Load the private key into a file.
  echo $GCP_CREDENTIALS | base64 --decode > ./.gcp/key.json

  # Change the permissions on the file to
  # be read-only for this user.
  chmod 400 ./.gcp/key.json
fi
0

If you're still having an issue running the app after following the buildpack instructions already mentioned in this article, try setting your Heroku environment variable GOOGLE_APPLICATION_CREDENTIALS to the full path instead.

GOOGLE_APPLICATION_CREDENTIALS = /app/google-credentials.json

This worked for me.

Previously, I could see that the google-credentials file was already generated by the buildpack (or .profile) but the Heroku app wasn't finding it, and giving errors as:

Error: Cannot find module './google-credentials.json'
Require stack:
- /app/server/src/config/firebase.js
- /app/server/src/middlewares/authFirebase.js
- /app/server/src/routes/v1/auth.route.js
- /app/server/src/routes/v1/index.js
- /app/server/src/app.js
- /app/server/src/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
    at Module.Hook._require.Module.require (/app/server/node_modules/require-in-the-middle/index.js:61:29)
    at require (internal/modules/cjs/helpers.js:93:18)
    at Object.<anonymous> (/app/server/src/config/firebase.js:3:24)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
0

I just have to add this tip, be careful on making GOOGLE_APPLICATION_CREDENTIALS variable in heroku dashborad, it caused me a day, if you have path like that for the credential file: server\credential.json , that will not work because using the backslash , so use slash instead / :
this will work as path (without "):
server/credential.json

0

The simplest way I've found is to

  1. Save the credentials as a string in a heroku ENV variable
  2. In you app, load them into a Ruby Tempfile
  3. Then have the GoogleDrive::Session.from_service_account_key load them from that temp file
require "tempfile"

...

google_credentials_tempfile = Tempfile.new("credentials.json")
google_credentials_tempfile.write(ENV["GOOGLE_CREDENTIALS_JSON"])
google_credentials_tempfile.rewind

session = GoogleDrive::Session.from_service_account_key(google_credentials_tempfile)

I have this in a heroku app and it works flawlessly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.