1

I have asked similar questions How to get ALL subdirectories, all levels deep except files in AWS S3 with python boto3and there are also others who have SIMILAR problems but this is more specific. I can get all sub-folders of any arbitrary depth from S3 using boto3 client (or aioboto3 for asycnhronous code) BUT it is very slow and it brings me back ALL the objects which I then filter using code like this:

    subfolders = set()
    prefix_tasks = [get_subfolders(bucket, prefix) for prefix in prefixes]
    try:

        for prefix_future in asyncio.as_completed(prefix_tasks):
            prefix_subfolders = await prefix_future
            subfolders.update(prefix_subfolders)

    except KeyError as exc:
        print(f"Scanning origin bucket failed due to: {exc}")
        raise exc

where my get_subfolders function is:

async def get_subfolders(self, bucket: str, prefix: str) -> Set[str]:

    subfolders = set()

    result = await self.s3_client.list_objects_v2(Bucket=bucket, Prefix=prefix)
    objects = result.get("Contents")
    subfolders.update(await self._get_paths_by_depth(objects=objects, depth=4))

    # Use next continuation token for pagination for truncated results.
    while result["IsTruncated"]:
        result = await self.s3_client.list_objects_v2(
            Bucket=bucket,
            Prefix=prefix,
            ContinuationToken=result["NextContinuationToken"],
        )
        objects = result.get("Contents")
        subfolders.update(await self._get_paths_by_depth(objects=objects, depth=4))

    return subfolders

and my get_paths_by_depth() function is :

    async def get_paths_by_depth(self, objects: dict, depth: int) -> Set[str]:
    subfolders = set()
    current_path = None
    try:
        # Get only paths with depth equal to 'depth' levels
        for bucket_object in objects:
            current_path = os.path.dirname(bucket_object["Key"])
            if current_path.count("/") == depth:
                subfolders.add(current_path)

    except Exception as exc:
        print(f"Getting subfolders failed due to error: {exc}")
        raise exc

    return subfolders

Any way to speed things up? I deally I would like to avoid bringing back all files and then filtering out the paths. Can I ask for paths of specific length right away?

2
  • Can you provide an example of your directory structure and what you are wanting as the result set (similar to how you did it in your other question)? Oct 16, 2021 at 2:41
  • yeah I actually found a way I ll post it below
    – KZiovas
    Oct 18, 2021 at 11:29

1 Answer 1

0

So the file structure I have is something like:

prefix/subfolder1/subfolder2/subfolder3/file1.txt
prefix/subfolder1/subfolder2/subfolder3/file2.json
prefix/subfolder4/subfolder5/file3.json
prefix/subfolder6/subfolder7/subfolder8/

and I only want to get the paths that end with at least one file in the above case I want to have at the end:

prefix/subfolder1/subfolder2/subfolder3/
prefix/subfolder4/subfolder5/

So far with the code I posted in the question I was looking at each single file in the bucket and keeping its path in a set. That worked but took way too long.

A much much faster way is to use the Delimiter parameter in the S3 request. Specifically I used the "." delimiter which changes the response of the s3_client and it includes all CommonPrefixes in the bucket which include a ".". Since all files include a "." I get all the common prefixes with a single request instead of checking each file. The new code is this :

async def get_subfolders(
    self, bucket: str, prefix: str, delimiter: str = "."
) -> Set[str]:

    subfolders = set()
    foldername = None
    try:
        paginator = self.s3_client.get_paginator("list_objects")
        async for result in paginator.paginate(
            Bucket=bucket, Prefix=prefix, Delimiter=delimiter
        ):
            for obj in result.get("CommonPrefixes", []):
                foldername = os.path.dirname(obj["Prefix"])
                # Get only paths with depth greater or equal than S3_FOLDERS_PATH_DEPTH
                if foldername.count("/") >= S3_FOLDERS_PATH_DEPTH:
                    subfolders.add(foldername)

    except Exception as exc:
        print(f"Getting subfolders failed due to error: {exc}")
        raise exc

    return subfolders

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.