30

I am not using Appengine.

I have a plain vanilla Django application running on a VM. I want to use Google Cloud Storage for serving my staticfiles, and also for uploading/serving my media files.

I have a bucket.

How do I link my Django application to my bucket? I've tried django-storages. That may work, but what do I have to do to prepare my bucket to be used by my django application? And what baseline configuration do I need in my Django settings?

Current settings:

# Google Cloud Storage
# http://django-storages.readthedocs.org/en/latest/backends/apache_libcloud.html
LIBCLOUD_PROVIDERS = {
    'google': {
        'type'  : 'libcloud.storage.types.Provider.GOOGLE_STORAGE',
        'user'  : <I have no idea>,
        'key'   : <ditto above>,
        'bucket': <my bucket name>,
    }
}

DEFAULT_LIBCLOUD_PROVIDER = 'google'
DEFAULT_FILE_STORAGE = 'storages.backends.apache_libcloud.LibCloudStorage'
STATICFILES_STORAGE = 'storages.backends.apache_libcloud.LibCloudStorage'

7 Answers 7

21

Django-storages has a backend for Google Cloud Storage, but it is not documented, I realised looking in the repo. Got it working with this setup:

DEFAULT_FILE_STORAGE = 'storages.backends.gs.GSBotoStorage'
GS_ACCESS_KEY_ID = 'YourID'
GS_SECRET_ACCESS_KEY = 'YourKEY'
GS_BUCKET_NAME = 'YourBucket'
STATICFILES_STORAGE = 'storages.backends.gs.GSBotoStorage'

To get YourKEY and YourID you should create Interoperability keys, in the settings tab.

Hope it helps and you don't have to learn it the hard way :)

Ah in case you haven't yet, the dependencies are:

pip install django-storages
pip install boto
4
  • I am getting GSResponseError: 401 Unauthorized any idea?
    – User707
    Mar 12, 2017 at 22:55
  • Why is STATICFILES_STORAGE = 'storages.backends.gs.GSBotoStorage' required? It causes browsable API to fail loading css files.
    – Kakaji
    May 15, 2017 at 7:08
  • I have a client_secret and a client_key generated from Oauth. Is that what I should be using for YourID and YourKEY?
    – Evan Zamir
    Aug 9, 2017 at 19:37
  • 1
    django-storages nowadays has gcloud backend using gcloud-python. It can be quite easily used to handle Django's static and media files. See my answer stackoverflow.com/a/50116653/5043802 and/or issue over at djang-storages: github.com/jschneier/django-storages/issues/491
    – elnygren
    May 1, 2018 at 12:47
16

Django-storages is, in fact, a viable alternative. You must be careful with it's Google Cloud backend though as the url() method it provides causes unnecessary HTTP calls to Google. (Django calls .url() when rendering static files, for example).

https://github.com/jschneier/django-storages/issues/491

settings.py

    DEFAULT_FILE_STORAGE = 'config.storage_backends.GoogleCloudMediaStorage'
    STATICFILES_STORAGE = 'config.storage_backends.GoogleCloudStaticStorage'
    GS_PROJECT_ID = '<google-cloud-project-id>'
    GS_MEDIA_BUCKET_NAME = '<name-of-static-bucket>'
    GS_STATIC_BUCKET_NAME = '<name-of-static-bucket>'
    STATIC_URL = 'https://storage.googleapis.com/{}/'.format(GS_STATIC_BUCKET_NAME)
    MEDIA_URL = 'https://storage.googleapis.com/{}/'.format(GS_MEDIA_BUCKET_NAME)

storage_backends.py

    """
    GoogleCloudStorage extensions suitable for handing Django's
    Static and Media files.

    Requires following settings:
    MEDIA_URL, GS_MEDIA_BUCKET_NAME
    STATIC_URL, GS_STATIC_BUCKET_NAME

    In addition to
    https://django-storages.readthedocs.io/en/latest/backends/gcloud.html
    """
    from django.conf import settings
    from storages.backends.gcloud import GoogleCloudStorage
    from storages.utils import setting
    from urllib.parse import urljoin


    class GoogleCloudMediaStorage(GoogleCloudStorage):
        """GoogleCloudStorage suitable for Django's Media files."""

        def __init__(self, *args, **kwargs):
            if not settings.MEDIA_URL:
                raise Exception('MEDIA_URL has not been configured')
            kwargs['bucket_name'] = setting('GS_MEDIA_BUCKET_NAME', strict=True)
            super(GoogleCloudMediaStorage, self).__init__(*args, **kwargs)

        def url(self, name):
            """.url that doesn't call Google."""
            return urljoin(settings.MEDIA_URL, name)


    class GoogleCloudStaticStorage(GoogleCloudStorage):
        """GoogleCloudStorage suitable for Django's Static files"""

        def __init__(self, *args, **kwargs):
            if not settings.STATIC_URL:
                raise Exception('STATIC_URL has not been configured')
            kwargs['bucket_name'] = setting('GS_STATIC_BUCKET_NAME', strict=True)
            super(GoogleCloudStaticStorage, self).__init__(*args, **kwargs)

        def url(self, name):
            """.url that doesn't call Google."""
            return urljoin(settings.STATIC_URL, name)

Note: authentication is handled by default via the GOOGLE_APPLICATION_CREDENTIALS environment variable.

https://cloud.google.com/docs/authentication/production#setting_the_environment_variable

14

May, 2022 Update:

With this instruction, you can connect your Django app to your bucket on GCS(Google Cloud Storage) and you can serve your static files and serve, upload and delete your media files.

For example, you have the bucket "my-django-bucket" on GCS:

enter image description here

And you have the service account "my-django-bucket-sa" then you need to copy(Ctrl+C) the email "my[email protected]":

enter image description here

Next, in Bucket details of the bucket "my-django-bucket", click on "PERMISSIONS" then "ADD":

enter image description here

Then, to enable the service account "my-django-bucket-sa" to have the full control of GCS resources, paste(Ctrl+V) the email "my[email protected]" to "New principals" then choose the role "Storage Admin" then click on "SAVE". *Choose other role by checking IAM roles for Cloud Storage if you don't want the role "Storage Admin" which can have the full control of GCS resources:

enter image description here

Next, to enable all users to view(read) files, type "allUsers" to “New principals” then choose the role "Storage Legacy Object Reader" then click on "SAVE":

enter image description here

Then, you should be asked as shown below so click on "ALLOW PUBLIC ACCESS":

enter image description here

Finally, you could add the role "Storage Admin" to the service account "my-django-bucket-sa" and the role "Storage Legacy Object Reader" to "allUsers":

enter image description here

Next, you need to download the private key of the service account "my-django-bucket-sa" in JSON so click on "Manage keys" from the 3 dots "⋮":

enter image description here

Then, click on "Create new key" from "ADD KEY":

enter image description here

Then, choose "JSON" then click on "CREATE"

enter image description here

Finally, you could download the private key of the service account "my-django-bucket-sa" in JSON "myproject-347313-020754294843.json":

enter image description here

Now, you have a django project and there are one settings folder "core" which has "static/core/core.js" and "settings.py" and one application folder "myapp" which has "static/myapp/myapp.css" as shown below:

enter image description here

Then next, you need to put "myproject-347313-020754294843.json" to the root django project directory where "db.sqlite3" and "manage.py" are:

enter image description here

Then, you better rename "myproject-347313-020754294843.json" to a shorter and reasonable name such as "gcpCredentials.json":

enter image description here

Next, you need to install "django-storages[google]" to connect to and communicate with "my-django-bucket" on GCS(Google Cloud Storage):

pip install django-storages[google]

By installing "django-storages[google]", you can get "django-storages" and other necessary packages as shown below:

"requirements.txt"

django-storages==1.12.3
cachetools==4.2.4
google-api-core==2.7.2
google-auth==2.6.5
google-cloud-core==2.3.0
google-cloud-storage==2.0.0
google-crc32c==1.3.0
google-resumable-media==2.3.2
googleapis-common-protos==1.56.0
protobuf==3.19.4
pyasn1==0.4.8
pyasn1-modules==0.2.8

Be careful, if you install "django-storages" without "[google]" as shown below:

pip install django-storages

You can only get "django-storages" as shown below.

"requirements.txt"

django-storages==1.12.3

Next, create "gcsUtils.py" in "core" folder where "settings.py" is:

enter image description here

Then, put this code below to "gcsUtils.py" to define "Static" and "Media" variables which each have a "GoogleCloudStorage" class object:

# "core/gcsUtils.py"

from storages.backends.gcloud import GoogleCloudStorage

Static = lambda: GoogleCloudStorage(location='static')
Media = lambda: GoogleCloudStorage(location='media')

Next, add this code below to "settings.py". *"STATICFILES_STORAGE" is like the conbination of "STATIC_ROOT" and "STATIC_URL" and "DEFAULT_FILE_STORAGE" is like the conbination of "MEDIA_ROOT" and "MEDIA_URL":

# "core/settings.py"

from google.oauth2 import service_account

# Set "static" folder
STATICFILES_STORAGE = 'core.gcsUtils.Static'

# Set "media" folder
DEFAULT_FILE_STORAGE = 'core.gcsUtils.Media'

GS_BUCKET_NAME = 'my-django-bucket'

# Add an unique ID to a file name if same file name exists
GS_FILE_OVERWRITE = False

GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    os.path.join(BASE_DIR, 'gcpCredentials.json'),
)

Then, run this command below:

python manage.py collectstatic

Now, "static" folder is created in "my-django-bucket":

enter image description here

And static files are collected from "admin" and "application" folders to "static" folder in "my-django-bucket":

enter image description here

And this is "myapp.css" in "myapp" folder:

enter image description here

But as you can see, static files are not collected from the settings folder "core" to "static" folder in "my-django-bucket":

enter image description here

Because "STATICFILES_STORAGE" can only collect static files from "admin" and "application" folders but not from other folders like the settings folder "core":

# "core/settings.py"

STATICFILES_STORAGE = 'core.gcsUtils.Static'

So, to collect static files from the settings folder "core" to "static" folder in "my-django-bucket":

enter image description here

You need to add "STATICFILES_DIRS" to "settings.py" as shown below:

# "core/settings.py"

# Collect static files from the settings folder
# "core" which is not "admin" and "application" folder
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'core/static'),
]

Then, this is the full code of "settings.py":

# "core/settings.py"

from google.oauth2 import service_account

# Collect static files from the settings folder
# "core" which is not "admin" and "application" folder
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'core/static'),
]

# Set "static" folder
STATICFILES_STORAGE = 'core.gcsUtils.Static'

# Set "media" folder
DEFAULT_FILE_STORAGE = 'core.gcsUtils.Media'

GS_BUCKET_NAME = 'my-django-bucket'

# Add an unique ID to a file name if same file name exists
GS_FILE_OVERWRITE = False

GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    os.path.join(BASE_DIR, 'gcpCredentials.json'),
)

Then again, run this command below:

python manage.py collectstatic

Then, static files is collected from the settings folder "core" to "static" folder in "my-django-bucket":

enter image description here

And this is "core.js" in "core" folder:

enter image description here

Next, this is the code for "myapp/models.py":

# "myapp/models.py"

from django.db import models

class Image(models.Model):
    image = models.ImageField(upload_to='images/fruits')
    
    def  __str__(self):
        return  str(self.image)

And this is the code for "myapp/admin.py":

# "myapp/admin.py"

from django.contrib import admin
from .models import Image

admin.site.register(Image)

Then, upload "orange.jpg":

enter image description here

Now, "media" folder is created in "my-django-bucket":

enter image description here

And "orange.jpg" is uploaded in "media/images/fruits":

enter image description here

And because "GS_FILE_OVERWRITE = False" is set in "settings.py":

# "core/settings.py"

# Add an unique ID to a file name if same file name exists
GS_FILE_OVERWRITE = False

If uploading the same name file "orange.jpg" again:

enter image description here

Then, the unique ID "_VPJxGBW" is added to "orange.jpg" to prevent file overwrite then, "orange_VPJxGBW.jpg" is uploaded as shown below:

enter image description here

Next, if there is "orange.jpg" in "media/images/fruits":

enter image description here

Then, update(change) "orange.jpg" to "apple.jpg" being uploaded:

enter image description here

Then, "apple.jpg" is uploaded in "media/images/fruits" but "orange.jpg" is still in "media/images/fruits" without deleted:

enter image description here

And if there is "orange.jpg" in "media/images/fruits":

enter image description here

Then, delete "orange.jpg":

enter image description here

But "orange.jpg" is still in "media/images/fruits" without deleted:

enter image description here

So, to delete uploaded files when they are updated(changed) and deleted, you need to install "django-cleanup":

pip install django-cleanup

Then, add it to the bottom of "INSTALLED_APPS" in "settings.py":

# "core/settings.py"

INSTALLED_APPS = (
    ...,
    'django_cleanup.apps.CleanupConfig', # Here
)

Then, if there is "orange.jpg" in "media/images/fruits":

enter image description here

Then, update(change) "orange.jpg" to "apple.jpg" being uploaded:

enter image description here

Then, "apple.jpg" is uploaded in "media/images/fruits" and "orange.jpg" is deleted from "media/images/fruits":

enter image description here

And if there is "orange.jpg" in "media/images/fruits":

enter image description here

Then, delete "orange.jpg":

enter image description here

Then, "orange.jpg" is deleted from "media/images/fruits":

enter image description here

Lastly, the GCS Bucket Settings which you have just set in "settings.py" as shown below work in both "DEBUG = True" and "DEBUG = False":

# "core/settings.py"

from google.oauth2 import service_account

# Collect static files from the settings folder
# "core" which is not "admin" and "application" folder
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'core/static'),
]

# Set "static" folder
STATICFILES_STORAGE = 'core.gcsUtils.Static'

# Set "media" folder
DEFAULT_FILE_STORAGE = 'core.gcsUtils.Media'

GS_BUCKET_NAME = 'my-django-bucket'

# Add an unique ID to a file name if same file name exists
GS_FILE_OVERWRITE = False

GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    os.path.join(BASE_DIR, 'gcpCredentials.json'),
)

And, there will be "STATIC_ROOT", "STATIC_URL", "MEDIA_ROOT", "MEDIA_URL" with the GCS Bucket Settings in "settings.py" as shown below. So, in this case, "STATIC_ROOT", "STATIC_URL", "MEDIA_ROOT" and "MEDIA_URL" don't work while the GCS Bucket Settings work communicating with "my-django-bucket" on GCS:

# "core/settings.py"

from google.oauth2 import service_account

STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATIC_URL = '/static/'
  
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'

'''GCS Bucket Settings Start'''
# Collect static files from the settings folder
# "core" which is not "admin" and "application" folder
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'core/static'),
]

# Set "static" folder
STATICFILES_STORAGE = 'core.gcsUtils.Static'

# Set "media" folder
DEFAULT_FILE_STORAGE = 'core.gcsUtils.Media'

GS_BUCKET_NAME = 'my-django-bucket'

# Add an unique ID to a file name if same file name exists
GS_FILE_OVERWRITE = False

GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    os.path.join(BASE_DIR, 'gcpCredentials.json'),
)
'''GCS Bucket Settings End'''

So, if you want "STATIC_ROOT", "STATIC_URL", "MEDIA_ROOT" and "MEDIA_URL" to work, just comment the GCS Bucket Settings then set "STATICFILES_DIRS" as shown below:

# "core/settings.py"

from google.oauth2 import service_account

# Collect static files from the settings folder
# "core" which is not "admin" and "application" folder
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'core/static'),
]

STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATIC_URL = '/static/'
  
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'

"""
'''GCS Bucket Settings Start'''
# Collect static files from the settings folder
# "core" which is not "admin" and "application" folder
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'core/static'),
]

# Set "static" folder
STATICFILES_STORAGE = 'core.gcsUtils.Static'

# Set "media" folder
DEFAULT_FILE_STORAGE = 'core.gcsUtils.Media'

GS_BUCKET_NAME = 'my-django-bucket'

# Add an unique ID to a file name if same file name exists
GS_FILE_OVERWRITE = False

GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    os.path.join(BASE_DIR, 'gcpCredentials.json'),
)
'''GCS Bucket Settings End'''
"""
6

So, this basically will work. (With this library and settings).

The trick to making it work, is knowing where to get the 'user' and 'key' parameters for libcloud.

On Google Cloud Console > Storage, click Settings. Then click on the right-hand tab called Interoperability. On that panel, is a lone button, which says something like Enable Interoperability. Click it.

Voila! You now have a username and key.


Note: Do not use django-storages from pypi. It has not been updated, and doesn't work with recent releases of Django.

Use this version:

pip install -e 'git+https://github.com/jschneier/django-storages.git#egg=django-storages'


Edit: If you want to use a reverse proxy, then you may consider my slightly modified version. https://github.com/jschneier/django-storages/compare/master...halfnibble:master

Description: Under certain circumstances, it may be necessary to load files using a reverse proxy. This could be used to alleviate cross-origin request errors.

This small PR allows the developer to set an optional LIBCLOUD_PROXY_URL in settings.py.

Example Usage

# Apache VirtualHost conf
ProxyPass /foo http://storage.googleapis.com
ProxyPassReverse /foo http://storage.googleapis.com


# settings.py
LIBCLOUD_PROXY_URL = '/foo/'
7
  • Further note. If you need secured file storage, then I strongly recommend AWS S3 with django_s3_storage. Unfortunately, there doesn't appear to be any Google Cloud Storage libraries that support access control for reading files.
    – Nostalg.io
    May 6, 2016 at 2:22
  • So if I understand correctly the django-storages will enable me to use ImageField with Google Cloud Storage as the backend. When I want to render the image in the browser (or an app), will I have access to an image url?
    – Evan Zamir
    Aug 2, 2017 at 17:21
  • @EvanZamir You are correct. ImageField should work just like FileField when it comes to saving to storage.
    – Nostalg.io
    Aug 2, 2017 at 17:41
  • Ok, so I am working with a strict app that only allows https content from the original site. Would I need to use a reverse proxy like this example? Apr 5, 2018 at 22:32
  • 1
    @ThatCampbellKid You absolutely could use a reverse proxy to achieve that goal. That is often why I use a reverse proxy.
    – Nostalg.io
    Apr 6, 2018 at 14:39
6

As the latest version, access key and key ID changed to service account file. And we want to use a bucket with 2 folders static and media like a local server. Below low the update configs:

Create a file like gcloud_storages.py:


"""
Modify django-storages for GCloud to set static, media folder in a bucket
"""
from django.conf import settings
from storages.backends.gcloud import GoogleCloudStorage


class GoogleCloudMediaStorage(GoogleCloudStorage):
    """
    GoogleCloudStorage suitable for Django's Media files.
    """

    def __init__(self, *args, **kwargs):
        kwargs['location'] = 'media'
        super(GoogleCloudMediaStorage, self).__init__(*args, **kwargs)


class GoogleCloudStaticStorage(GoogleCloudStorage):
    """
    GoogleCloudStorage suitable for Django's Static files
    """

    def __init__(self, *args, **kwargs):
        kwargs['location'] = 'static'
        super(GoogleCloudStaticStorage, self).__init__(*args, **kwargs)

Use location argument to set the location of static, media files in bucket.

In settings.py

from google.oauth2 import service_account
...

GOOGLE_APPLICATION_CREDENTIALS = '/path/service-account.json'
DEFAULT_FILE_STORAGE = 'app.gcloud_storages.GoogleCloudMediaStorage'
STATICFILES_STORAGE = 'app.gcloud_storages.GoogleCloudStaticStorage'
GS_BUCKET_NAME = 'name-of-bucket'
GS_PROJECT_ID = 'project-id'
GS_DEFAULT_ACL = 'publicRead'
GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    GOOGLE_APPLICATION_CREDENTIALS
)
1

Since I cannot comment on Alan Wagner's answer, here is an addition.

If you are using python3, you may get this error,

...
ImportError: No module named 'google_compute_engine'

If so, you will need to install google-compute-engine. The /etc/boto.cfg file tells python to use the 2.7 version of the library. You will have to run this next line to regenerate /etc/boto.cfg.

python3 -c "from google_compute_engine.boto.boto_config import BotoConfig; BotoConfig()"

Another error you may hit,

...
File "/app/venv/lib/python3.4/site-packages/boto/gs/connection.py", line 95, in create_bucket
    data=get_utf8_value(data))
File "/app/venv/lib/python3.4/site-packages/boto/s3/connection.py", line 656, in make_request
    auth_path = self.calling_format.build_auth_path(bucket, key)
File "/app/venv/lib/python3.4/site-packages/boto/s3/connection.py", line 94, in build_auth_path
    path = '/' + bucket
TypeError: Can't convert 'bytes' object to str implicitly

I made a pull request to fix this. You may use my repo as a pip dependency if you wish until it gets merged.

I will try to keep this repo up to date. I have set the default develop branch as protected. I am the only one who can commit/approve merge requests. I have also only made one commit.

You will have to install google-compute-engine and run that line above before you can instalkl/build my boto repo.

1

I have detailed my step by step process on another thread

Here are my main references:

I used the following packages:

pip3 install django-storages # https://pypi.org/project/django-storages/
pip3 install google-cloud-storage # https://pypi.org/project/google-cloud-storage/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.