1

I am writing a programme that goes through a large number of storage objects using Google's google-api-python-client, but I'm wondering if there is a more efficient way I'm overlooking?

At the moment we have a total of around 35_000 objects and I'm iterating through each one, one-by-one.

So first I'm getting a list of buckets:

from googleapiclient import discovery
from oauth2client.service_account import ServiceAccountCredentials

credentials = ServiceAccountCredentials.from_json_keyfile_name('keyfile')
session = discovery.build('storage', 'v1', credentials=credentials)

def list_buckets():
    all_buckets = session.buckets().list(project='project').execute()
    return [bucket['name'] for bucket in all_buckets['items']]

I then iterate over my bucket list and return a list of objects:

for bucket in list_buckets():
    all_objects = session.objects().list(bucket=bucket)

I then append each name and bucket_name from the above result to a namedtuple, which I loop over one more time:

for obj in all_objects:
    bucket_session = session.objectAccessControls().list(
        bucket=obj.bucket,
        object=obj.name).execute()

The above gets appended to a list and then I finally iterate over the list and check permissions.

As you can imagine this takes some time with objects numbering in the tens of thousands, so I was wondering whether anyone has done this in a more efficient way? Or perhaps I should be using the more modern google-cloud library?

1 Answer 1

1

If your goal is to process the access control list for each object, the API supports returning the full access controls for the object as part of the bucket listing. This can save making a round-trip to the API to retrieve each object's access controls. It looks like you could do this by setting projection to "full" in google-cloud-python's list_blobs function.

gsutil -p your-project ls -L gs://*/** would also give you a text dump of all the access controls for your objects in your-project.

Finally, to make auditing easier in the future, you might consider managing access at the bucket or project level using IAM, though it depends on your use case.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.