import boto3
import os
client = boto3.client('connect')
s3 = boto3.resource(
service_name='s3',
region_name='us-west-2',
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key
)
for my_bucket_object in s3.Bucket("my_bucket").objects.filter(Prefix="user/folder/"):
s3.Object(my_bucket_object.bucket_name, my_bucket_object.key).download_file(f'./aws/{my_bucket_object.key}')
- Without iteration, but similar code, I can successfully download individual files.
- Without downloading, printing the bucket keys shows normal outputs
However, when I iterate over multiple files, and use the key as input for download_file, I get the following error message. Target key's name seems to be changing?
FileNotFoundError: [Errno 2] No such file or directory: './aws/user/folder\.7g4DBa9A'
I have the following two questions:
- How can I prevent this from happening and download the files?
- Is there a way to separate file names from "subdirectories" (I realize AWS doesn't use those, but keys contain directory/file-like names separated only by "/", I would like to separate those for saving purposes)
===========================================================================
Found the answer thanks to Marcin's comment.
After iteratively printing all the outputs, it seemed the first one was the "folder", which translated to strange names when downloading.
ie.
user/folder/
user/folder/file1
user/folder/file2
etc.
Thus, ignoring that first iteration was able to solve it.
for obj in my_bucket.objects.filter(Prefix=prefix):
output_file = obj.key.split('/')[-1]
if output_file == "":
continue
else:
s3.Object(bucket_name=my_bucket.name, key=my_bucket_object.key).download_file(arbitrary output path)