Short answer:
Use Delimiter='/'
. This avoids doing a recursive listing of your bucket. Some answers here wrongly suggest doing a full listing and using some string manipulation to retrieve the directory names. This could be horribly inefficient. Remember that S3 has virtually no limit on the number of objects a bucket can contain. So, imagine that, between bar/
and foo/
, you have a trillion objects: you would wait a very long time to get ['bar/', 'foo/']
.
Use Paginators
. For the same reason (S3 is an engineer's approximation of infinity), you must list through pages and avoid storing all the listing in memory. Instead, consider your "lister" as an iterator, and handle the stream it produces.
Use boto3.client
, not boto3.resource
. The resource
version doesn't seem to handle well the Delimiter
option. If you have a resource, say a bucket = boto3.resource('s3').Bucket(name)
, you can get the corresponding client with: bucket.meta.client
.
Long answer:
The following is an iterator that I use for simple buckets (no version handling).
import os
import boto3
from collections import namedtuple
from operator import attrgetter
S3Obj = namedtuple('S3Obj', ['key', 'mtime', 'size', 'ETag'])
def s3list(bucket, path, start=None, end=None, recursive=True, list_dirs=True,
list_objs=True, limit=None):
"""
Iterator that lists a bucket's objects under path, (optionally) starting with
start and ending before end.
If recursive is False, then list only the "depth=0" items (dirs and objects).
If recursive is True, then list recursively all objects (no dirs).
Args:
bucket:
a boto3.resource('s3').Bucket().
path:
a directory in the bucket.
start:
optional: start key, inclusive (may be a relative path under path, or
absolute in the bucket)
end:
optional: stop key, exclusive (may be a relative path under path, or
absolute in the bucket)
recursive:
optional, default True. If True, lists only objects. If False, lists
only depth 0 "directories" and objects.
list_dirs:
optional, default True. Has no effect in recursive listing. On
non-recursive listing, if False, then directories are omitted.
list_objs:
optional, default True. If False, then directories are omitted.
limit:
optional. If specified, then lists at most this many items.
Returns:
an iterator of S3Obj.
Examples:
# set up
>>> s3 = boto3.resource('s3')
... bucket = s3.Bucket('bucket-name')
# iterate through all S3 objects under some dir
>>> for p in s3list(bucket, 'some/dir'):
... print(p)
# iterate through up to 20 S3 objects under some dir, starting with foo_0010
>>> for p in s3list(bucket, 'some/dir', limit=20, start='foo_0010'):
... print(p)
# non-recursive listing under some dir:
>>> for p in s3list(bucket, 'some/dir', recursive=False):
... print(p)
# non-recursive listing under some dir, listing only dirs:
>>> for p in s3list(bucket, 'some/dir', recursive=False, list_objs=False):
... print(p)
"""
kwargs = dict()
if start is not None:
if not start.startswith(path):
start = os.path.join(path, start)
# note: need to use a string just smaller than start, because
# the list_object API specifies that start is excluded (the first
# result is *after* start).
kwargs.update(Marker=__prev_str(start))
if end is not None:
if not end.startswith(path):
end = os.path.join(path, end)
if not recursive:
kwargs.update(Delimiter='/')
if not path.endswith('/'):
path += '/'
kwargs.update(Prefix=path)
if limit is not None:
kwargs.update(PaginationConfig={'MaxItems': limit})
paginator = bucket.meta.client.get_paginator('list_objects')
for resp in paginator.paginate(Bucket=bucket.name, **kwargs):
q = []
if 'CommonPrefixes' in resp and list_dirs:
q = [S3Obj(f['Prefix'], None, None, None) for f in resp['CommonPrefixes']]
if 'Contents' in resp and list_objs:
q += [S3Obj(f['Key'], f['LastModified'], f['Size'], f['ETag']) for f in resp['Contents']]
# note: even with sorted lists, it is faster to sort(a+b)
# than heapq.merge(a, b) at least up to 10K elements in each list
q = sorted(q, key=attrgetter('key'))
if limit is not None:
q = q[:limit]
limit -= len(q)
for p in q:
if end is not None and p.key >= end:
return
yield p
def __prev_str(s):
if len(s) == 0:
return s
s, c = s[:-1], ord(s[-1])
if c > 0:
s += chr(c - 1)
s += ''.join(['\u7FFF' for _ in range(10)])
return s
Test:
The following is helpful to test the behavior of the paginator
and list_objects
. It creates a number of dirs and files. Since the pages are up to 1000 entries, we use a multiple of that for dirs and files. dirs
contains only directories (each having one object). mixed
contains a mix of dirs and objects, with a ratio of 2 objects for each dir (plus one object under dir, of course; S3 stores only objects).
import concurrent
def genkeys(top='tmp/test', n=2000):
for k in range(n):
if k % 100 == 0:
print(k)
for name in [
os.path.join(top, 'dirs', f'{k:04d}_dir', 'foo'),
os.path.join(top, 'mixed', f'{k:04d}_dir', 'foo'),
os.path.join(top, 'mixed', f'{k:04d}_foo_a'),
os.path.join(top, 'mixed', f'{k:04d}_foo_b'),
]:
yield name
with concurrent.futures.ThreadPoolExecutor(max_workers=32) as executor:
executor.map(lambda name: bucket.put_object(Key=name, Body='hi\n'.encode()), genkeys())
The resulting structure is:
./dirs/0000_dir/foo
./dirs/0001_dir/foo
./dirs/0002_dir/foo
...
./dirs/1999_dir/foo
./mixed/0000_dir/foo
./mixed/0000_foo_a
./mixed/0000_foo_b
./mixed/0001_dir/foo
./mixed/0001_foo_a
./mixed/0001_foo_b
./mixed/0002_dir/foo
./mixed/0002_foo_a
./mixed/0002_foo_b
...
./mixed/1999_dir/foo
./mixed/1999_foo_a
./mixed/1999_foo_b
With a little bit of doctoring of the code given above for s3list
to inspect the responses from the paginator
, you can observe some fun facts:
The Marker
is really exclusive. Given Marker=topdir + 'mixed/0500_foo_a'
will make the listing start after that key (as per the AmazonS3 API), i.e., with .../mixed/0500_foo_b
. That's the reason for __prev_str()
.
Using Delimiter
, when listing mixed/
, each response from the paginator
contains 666 keys and 334 common prefixes. It's pretty good at not building enormous responses.
By contrast, when listing dirs/
, each response from the paginator
contains 1000 common prefixes (and no keys).
Passing a limit in the form of PaginationConfig={'MaxItems': limit}
limits only the number of keys, not the common prefixes. We deal with that by further truncating the stream of our iterator.
/
to get subfolders