38

How can I recursively list all all the contents of a bucket using the AWS CLI similar to using find . on Unix.

aws s3 ls s3://MyBucket --recursive complains with unknown option.

http://docs.aws.amazon.com/cli/latest/reference/s3/index.html#directory-and-s3-prefix-operations claims that --recursive is a valid parameter.

5 Answers 5

63

aws s3 ls s3://MyBucket --recursive works fine for me.

Try updating your AWS CLI. My version is aws-cli/1.6.2

aws --version

1
  • Original question was from a while ago with version 1.3.X Nov 21, 2014 at 20:22
4

With recent AWS CLI versions, --recursive option is supported.

You can list recursively all the files under a bucket named MyBucket using following command:

aws s3 ls s3://MyBucket/ --recursive

You can list recursively all the files under a folder named MyFolder in the bucket, using following command:

aws s3 ls s3://MyBucket/MyFolder/ --recursive

As @Francisco Cardoso said, the final / is very important. It allows to list the content of the folder instead of the folder itself

For more information, see: https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html

1
  • 1
    That final "/" after MyFolder is very important Mar 30, 2022 at 17:07
1

I am not able to interpret the link you referred properly: http://docs.aws.amazon.com/cli/latest/reference/s3/index.html#directory-and-s3-prefix-operations

However, I was able to make --recursive option work with respect to this link: http://docs.aws.amazon.com/cli/latest/reference/s3/index.html#single-local-file-and-s3-object-operations

as per this link, cp, mv and rm supports --recursive option.

The one that you are trying is ls.

I tried using cp and rm with --recursive option and it is working fine.

1

You can not list recursively all the contents of a bucket via -

aws s3 ls s3://MyBucket

To list object from a folder you need to execute command as -

aws s3 ls s3://MyBucket/MyFolder/

This above command lists object that reside inside folder named MyFolder.

To get an objects list from such a logical hierarchy from Amazon S3, you need specify the full key name for the object in the GET operation.

--recursive Command is performed on allfiles or objects under the specified directory or prefix.

Thanks

3
  • 1
    This is no longer true. Jan 9, 2018 at 20:46
  • best to delete obsolete info Apr 20, 2018 at 17:06
  • 1
    Is there a way to get only file names and not timestamps, etc? Jun 8, 2020 at 14:42
0

Below one line bash script is able to perform:- how to list all S3 buckets with their objects recursively, list bucket name and count objects also.

/usr/bin/sudo /usr/local/bin/aws s3 ls |awk '{print $NF}'| while read l;do echo -e "#######---$l objects---##########\n\n";/usr/bin/sudo /usr/local/bin/aws s3 ls $l|nl;done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.