91

In Google Colaboratory, I can install a new library using !pip install package-name. But when I open the notebook again tomorrow, I need to re-install it every time.

Is there a way to install a library permanently? No need to spend time installing every time to use?

4 Answers 4

97

Yes. You can install the library in Google Drive. Then add the path to sys.path.

import os, sys
from google.colab import drive
drive.mount('/content/drive')
nb_path = '/content/notebooks'
os.symlink('/content/drive/My Drive/Colab Notebooks', nb_path)
sys.path.insert(0,nb_path)

Then you can install a library, for example, jdc, and specify the target.

!pip install --target=$nb_path jdc

Later, when you run the notebook again, you can skip the !pip install line. You can just import jdc and use it. Here's an example notebook.

https://colab.research.google.com/drive/1KpMDi9CjImudrzXsyTDAuRjtbahzIVjq

BTW, I really like jdc's %%add_to. It makes working with a big class much easier.

12
  • Your idea is very interesting and useful. However I should make authorization every time. I would like to do this process only once. Do you know a solution for this also? If yes, I will ask it in a different post.
    – vesszabo
    Aug 29, 2019 at 8:07
  • 1
    @vesszabo I guess it's possible. Instead of GDrive mount, you need to use gcsfuse to mount Google Cloud Storage bucket using embeded credential in your notebook (using %%writefile). Though I never try it.
    – korakot
    Aug 29, 2019 at 10:11
  • 1
    who is jdc and what does %%add_to cell magic do?
    – vwvan
    Apr 15, 2022 at 11:14
  • @vwvan jdc is a library. It allows %%add_to that add a new method to an existing class. See the notebook I linked.
    – korakot
    Apr 15, 2022 at 12:15
  • 1
    Hi, I am getting all the time the error No such file or directory: '/content/notebooks/...' has anyone any idea on how to solve it?
    – MarinerZZ
    Apr 20 at 10:14
19

If you want a no-authorization solution. You can use mounting with gcsfuse + service-account key embedded in your notebook. Like this:

# first install gcsfuse
%%capture
!echo "deb http://packages.cloud.google.com/apt gcsfuse-bionic main" > /etc/apt/sources.list.d/gcsfuse.list
!curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
!apt update
!apt install gcsfuse

Then get your service account credential from google cloud console and embed it in the notebook

%%writefile /key.json
{
  "type": "service_account",
  "project_id": "kora-id",
  "private_key_id": "xxxxxxx",
  "private_key": "-----BEGIN PRIVATE KEY-----\nxxxxxxx==\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "100380920993833371482",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/colab-7%40kora-id.iam.gserviceaccount.com"
}

Then set environment to look for this credential file

%env GOOGLE_APPLICATION_CREDENTIALS=/key.json

You must then create (or have it already) a gcs bucket. And mount it to a made-up directory.

!mkdir /content/my-bucket
!gcsfuse my-bucket /content/my-bucket

Then finally, install the library there. Like my above answer.

import sys
nb_path = '/content/my-bucket'
sys.path.insert(0, nb_path)
# Do this just once
!pip install --target=$nb_path jdc

You can now import jdc without !pip install it next time.

4
  • 3
    It's a few steps to get the json credential. Read this cloud.google.com/iam/docs/…
    – korakot
    Aug 29, 2019 at 14:22
  • @KorakotChaovavanich, I successfully followed all the step. But, the folder my-bucket is not around in new runtime. May I know where I do wrong?
    – mpx
    Oct 13, 2019 at 1:56
  • You must create a new gcs bucket here.(don't use my-bucket, use your own name) console.cloud.google.com/storage/browser
    – korakot
    Oct 13, 2019 at 14:59
  • Please add to the answer a link or a quick guide for how to create that json. Its definitely not trivial Nov 28, 2021 at 10:54
1

In case you need to install multiple libraries here is a snippet:

def install_library_to_drive(libraries_list):
  """ Install library on gdrive. Run this only once. """
  drive_path_root = 'path/to/mounted/drive/directory/where/you/will/install/libraries'
  for lib in libraries_list:
    drive_path_lib = drive_path_root + lib
    !pip install -q $lib --target=$drive_path_lib
    sys.path.insert(0, drive_path_lib)

def load_library_from_drive(libraries_list):
""" Technically, it just appends install dir to a sys.path """
  drive_path_root = 'path/to/mounted/drive/directory/where/you/will/install/libraries'
  for lib in libraries_list:
    drive_path_lib = drive_path_root + lib
    sys.path.insert(0, drive_path_lib)

libraries_list = ["torch", "jsonlines", "transformers"] # list your libraries
install_library_to_drive(libraries_list) # Run this just once
load_library_from_drive(libraries_list)
1

I have installed libraries permanently in google colab using a virtual environment. Use this blog as reference https://netraneupane.medium.com/how-to-install-libraries-permanently-in-google-colab-fb15a585d8a5

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.