5

I have a need to retrieve files in my web application that are greater than 64KB in size. Right now, in v1beta1 of the JSON API, Google is only allowing uploads/downloads of 64KB in size via their JSON API. I've figured out how to upload files over 64KB in size using a "resumable" upload (not via the interface v1beta1 provides in its JSON API, but manually).

What I can't figure out is a good way to download. Right now, I make the ACL public for the object I want to download, download the file, and then remove the public ACL on the object. This is not only inefficient, it is not very clean. Is there a better method I could use or am I stuck until Google provides a better means in a future version of their API?

Background Information I am writing a GAE application and I know of the google.appengine.api.files interface. Unfortunately this does not work on live buckets while using the local dev environment, and for testing purposes, my team and I need a way to test development locally (too cumbersome to deploy to GAE among other limiting/security factors). We can interact with all other APIs except for Cloud Storage, so I'm writing a class that will use either the JSON API or AppEngine's files interface when reading/writing/deleting from Cloud Storage. I got a working implementation, but I'm unhappy with the way I retrieve files.

Clarification from comment below: We are downloading large amounts of information, massaging it, and storing it into Cloud Storage for consumption into BigQuery. We need to use live buckets from the dev environment because if we don't, BigQuery won't be able to consume data we want to test. No need to serve these files, just process them

Solution from comment on accepted Answer below: I was able to reuse my authenticated httplib2 object from my code that interacts with the JSON API to do an authenticated GET request against the https://{bucket_name}.storage.googleapis.com/{object_name} URL endpoint, adding only the Content-Length: 0 and x-goog-api-version: 2 headers.

4
  • It's not clear from the question if you are using GAE or not? If so there are some alternatives depending your use case. Jan 24, 2013 at 19:58
  • My apologies, added some background information
    – someone1
    Jan 24, 2013 at 20:13
  • What is your use case? Do you need to serve the files or read them and proccess them? Also, why would you want to access live buckets from your dev environment? Jan 24, 2013 at 20:19
  • We are downloading large amounts of information, massaging it, and storing it into Cloud Storage for consumption into BigQuery. We need to use live buckets from the dev environment because if we don't, BigQuery won't be able to consume data we want to test. No need to serve these files, just process them.
    – someone1
    Jan 24, 2013 at 20:23

1 Answer 1

1

For downloads, the XML API is the simplest way to do an authenticated download. There are multiple ways to do authenticated downloads, as detailed on this page: https://developers.google.com/storage/docs/authentication

For a simple fetch operation, the name "XML API" could be considered misleading -- you don't need to produce any XML. If you experience an error, it will be represented as XML, but in the simplest case of a successful file download, no XML is actually involved.

6
  • It would be my preference to utilize the JSON API. I use it to communicate with a number of other Google APIs so a lot of the code is already written and easily portable to interact with it. There is also a built in library that authenticates to Google for me (AppAssertionCredentials) on production. Creating and managing a GAE-friendly way to communicate with the XML API would be somewhat cumbersome.
    – someone1
    Jan 24, 2013 at 23:49
  • Fair enough. Let me do a little research.
    – Benson
    Jan 25, 2013 at 1:16
  • Ideally I'd like for it to work like the Drive API. When I get a list of files or a specific file, I get downloadURLs/ExportLinks that I can make authenticated GET calls to download the resource. I already have an oauth2 token... maybe I can re-use that token with the XML API and ping a gateway with the bucket/object information to grab the data.... May have found my answer: developers.google.com/storage/docs/reference-methods#getobject
    – someone1
    Jan 25, 2013 at 1:41
  • Yeah, that's what I was suggesting. If you have an oauth2 token you should be able to just do a regular old GET with the appropriate oauth2 header. That's what we call the "XML API", because other things in that family of URLs accept or return XML.
    – Benson
    Jan 25, 2013 at 1:44
  • I was worried I would have to start producing/consuming XML so I didn't give the API a good look, my apologies. I was able to reuse my authenticated httplib2 object from my code that interacts with the JSON API to do an authenticated GET request against the https://{bucket_name}.storage.googleapis.com/{object_name} URL endpoint, adding only the Content-Length: 0 and x-goog-api-version: 2 headers. Thanks a bunch :)!
    – someone1
    Jan 25, 2013 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.