9

I cannot find a way to to write a data set from my local machine into the google cloud storage using python. I have researched a a lot but didn't find any clue regarding this. Need help, thanks

2

6 Answers 6

18

Quick example, using the google-cloud Python library:

from google.cloud import storage

def upload_blob(bucket_name, source_file_name, destination_blob_name):
  """Uploads a file to the bucket."""
  storage_client = storage.Client()
  bucket = storage_client.get_bucket(bucket_name)
  blob = bucket.blob(destination_blob_name)

  blob.upload_from_filename(source_file_name)

  print('File {} uploaded to {}.'.format(
      source_file_name,
      destination_blob_name))

More examples are in this GitHub repo: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/storage/cloud-client

4
  • What do you mean by a write? Aug 20, 2018 at 7:30
  • 1
    base on this link cloud.google.com/appengine/docs/standard/python/… (from what I understood) this will allow you to create the file in the bucket, instead of uploading
    – Manza
    Aug 20, 2018 at 11:13
  • Creating a file by specifying its contents is the same operation as uploading a new file. It's just called a different name here. Aug 20, 2018 at 21:30
  • my bad, I though in the upload you create a file and then upload, in the other i though you actually create the file directly in the bucket folder, for hence never in your server, I am having issues as appengine doesnt allows me to create files so i am trying to find that solution
    – Manza
    Aug 20, 2018 at 23:29
5

When we want to write a string to a GCS bucket blob, the only change necessary is using blob.upload_from_string(your_string) rather than blob.upload_from_filename(source_file_name):

from google.cloud import storage

def write_to_cloud(your_string):
    client = storage.Client()
    bucket = client.get_bucket('bucket123456789')
    blob = bucket.blob('PIM.txt')
    blob.upload_from_string(your_string)
4

In the earlier answers, I still miss the easiest way, using the open() method.

You can use the blob.open() as follows:

from google.cloud import storage
    
def write_file():
    client = storage.Client()
        bucket = client.get_bucket('bucket-name')
        blob = bucket.blob('path/to/new-blob-name.txt') 
        ## Use bucket.get_blob('path/to/existing-blob-name.txt') to write to existing blobs
        with blob.open(mode='w') as f:
            for line in object: 
                f.write(line)

You can find more examples and snippets here: https://github.com/googleapis/python-storage/tree/main/samples/snippets

3
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials

credentials = GoogleCredentials.get_application_default()
service = discovery.build('storage', 'v1', credentials=credentials)
filename = 'file.csv'
bucket = 'Your bucket name here'
body = {'name': 'file.csv'}    
req = service.objects().insert(bucket=bucket, body=body, media_body=filename)
resp = req.execute()
2
  • that looks like an upload not a write?
    – Manza
    Aug 18, 2018 at 1:47
  • 1
    @Manza isn't the same ? Sep 10, 2020 at 1:54
0
from google.cloud import storage

def write_to_cloud(buffer):
    client = storage.Client()
    bucket = client.get_bucket('bucket123456789')
    blob = bucket.blob('PIM.txt')
    blob.upload_from_file(buffer)

While Brandon's answer indeed gets the file to Google cloud, it does this by uploading the file, as opposed to writing the file. This means that the file needs to exist on your disk before you upload it to the cloud.

My proposed solution uses an "in-memory" payload (the buffer parameter) which is then written to cloud. To write the content you need to use upload_from_file instead of upload_from_filename, everything else being the same.

0
0
import logging
L = logging.getLogger(__name__)
from google.cloud import storage

def write_file(bucket_name, source_string, destination_blob_name):        
    storage_client = storage.Client()
    bucket = storage_client.get_bucket(bucket_name)
    if not bucket.blob(destination_blob_name).exists():
        L.warn(f"Blob {destination_blob_name} does not exist.")
        print(f"Blob {destination_blob_name} does not exist.")
        print(f"Creating blob {destination_blob_name}.")
        blob = bucket.blob(destination_blob_name)
    else:
        print(f"Blob {destination_blob_name} exists.")
    with blob.open(mode='w') as f:
        for line in source_string:
            f.write(line)
1
  • Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Feb 22 at 17:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.