189

When I log to my S3 console I am unable to download multiple selected files (the WebUI allows downloads only when one file is selected):

https://console.aws.amazon.com/s3

Is this something that can be changed in the user policy or is it a limitation of Amazon?

2
  • Please watch this video available on youtube. This guy has explained in a easy way. youtube.com/watch?v=fStDrddmi5Y Commented Feb 17, 2020 at 12:55
  • 5
    As @Aur Saraf suggests, once you have multiple selected, you can click "open" instead of "Download" to download them, it'll open multiple tabs and each tab will be a new download request. worked like a charm for me. I think given he gives a real solution to download multiple files using the s3 console, I think his answer should be selected as correct. all others are just suggestions to use aws client which I figure most people know how to do this. worse, the actual selected answer suggests there is no solution which is not true. Commented Jan 4, 2022 at 5:58

17 Answers 17

252

It is not possible through the AWS Console web user interface. But it's a very simple task if you install AWS CLI. You can check the installation and configuration steps on Installing in the AWS Command Line Interface

After that you go to the command line:

aws s3 cp --recursive s3://<bucket>/<folder> <local_folder> 

This will copy all the files from given S3 path to your given local path.

7
  • 19
    FYI: an S3 path starts with "s3://" as in s3://my-unique-bucket-name/one-folder/another-folder.
    – Marek P
    Commented May 21, 2018 at 20:38
  • 5
    What if I want to download just 10GB of data from a folder in a bucket, I have ~100GB of data in a folder btw. Is it possible to just get 10GB of data out of 100GB?
    – shammery
    Commented Nov 24, 2018 at 6:32
  • To get the S3 Path, go to the folder you want to download in the S3 Console, select a single file and click the button "Copy Path". Modifying the http URI that it provides does not work. Commented Apr 15, 2019 at 22:47
  • aws s3 cp --recursive s3://onlineneta/videos /var/www/html/video/ this command dose not work can you tell me why ? it give me error live fatal error: Unable to locate credentials Commented May 24, 2021 at 12:37
  • @BhavikHirani - please do aws configure to setup credentials
    – Abhay
    Commented Aug 16, 2021 at 9:44
80

Selecting a bunch of files and clicking Actions->Open opened each in a browser tab, and they immediately started to download (6 at a time).

Screenshot

9
  • 17
    Works for far more than 6 files in Chrome on Windows 10. This is the correct answer imho (what are AWS playing at with their UI design?) Commented May 3, 2019 at 10:40
  • 1
    I guess Chrome had the 6 file limit on my computer.
    – Aur Saraf
    Commented May 4, 2019 at 15:54
  • 2
    Downloaded 46 CSVs at once using this tip, thanks! (Safari on MacOS 11.0)
    – Mr Rabbit
    Commented Apr 23, 2021 at 15:00
  • 13
    This action is no longer available for multiple files.
    – bob
    Commented Nov 7, 2021 at 22:06
  • 2
    @bob As of now (08/2023) it was moved out of the menu and is a button beside the Download button that has the same effect when multiple files are selected.
    – Sherwin F
    Commented Aug 31, 2023 at 20:36
60

If you use AWS CLI, you can use the exclude along with --include and --recursive flags to accomplish this

aws s3 cp s3://path/to/bucket/ . --recursive --exclude "*" --include "things_you_want"

Eg.

--exclude "*" --include "*.txt"

will download all files with .txt extension. More details - https://docs.aws.amazon.com/cli/latest/reference/s3/

2
  • 4
    The order --exclude , then --include should not be changed
    – TamizhK
    Commented Apr 29, 2021 at 7:03
  • 1
    Also good to note that the --recursive flag is required. Or it will not work.
    – Akaisteph7
    Commented Jun 27, 2022 at 15:36
10

If you have Visual Studio with the AWS Explorer extension installed, you can also browse to Amazon S3 (step 1), select your bucket (step 2), select all the files you want to download (step 3) and right click to download them all (step 4).

enter image description here

2
  • 1
    Selecting multiple S3 files in VSCode from the AWS Explorer extension appears to no longer be possible.
    – LWSChad
    Commented Jul 21, 2023 at 14:48
  • 1
    @LWSChad That's Visual Studio, not VS Code. Commented Feb 8 at 1:28
8

I believe it is a limitation of the AWS console web interface, having tried (and failed) to do this myself.

Alternatively, perhaps use a 3rd party S3 browser client such as http://s3browser.com/

4

The S3 service has no meaningful limits on simultaneous downloads (easily several hundred downloads at a time are possible) and there is no policy setting related to this... but the S3 console only allows you to select one file for downloading at a time.

Once the download starts, you can start another and another, as many as your browser will let you attempt simultaneously.

3

In case someone is still looking for an S3 browser and downloader I have just tried Fillezilla Pro (it's a paid version). It worked great.

I created a connection to S3 with Access key and secret key set up via IAM. Connection was instant and downloading of all folders and files was fast.

2

I wrote a simple shell script to download NOT JUST all files but also all versions of every file from a specific folder under AWS s3 bucket. Here it is & you may find it useful

# Script generates the version info file for all the 
# content under a particular bucket and then parses 
# the file to grab the versionId for each of the versions
# and finally generates a fully qualified http url for
# the different versioned files and use that to download 
# the content.

s3region="s3.ap-south-1.amazonaws.com"
bucket="your_bucket_name"
# note the location has no forward slash at beginning or at end
location="data/that/you/want/to/download"
# file names were like ABB-quarterly-results.csv, AVANTIFEED--quarterly-results.csv
fileNamePattern="-quarterly-results.csv"

# AWS CLI command to get version info
content="$(aws s3api list-object-versions --bucket $bucket --prefix "$location/")"
#save the file locally, if you want
echo "$content" >> version-info.json
versions=$(echo "$content" | grep -ir VersionId  | awk -F ":" '{gsub(/"/, "", $3);gsub(/,/, "", $3);gsub(/ /, "", $3);print $3 }')
for version in $versions
do
    echo ############### $fileId ###################
    #echo $version
    url="https://$s3region/$bucket/$location/$fileId$fileNamePattern?versionId=$version"
    echo $url
    content="$(curl -s "$url")"
    echo "$content" >> $fileId$fileNamePattern-$version.csv
    echo ############### $i ###################
done
2

Using AWS CLI, I ran all the downloads in the background using "&" and then waited on all the pids to complete. It was amazingly fast. Apparently the "aws s3 cp" knows to limit the number of concurrent connections because it only ran 100 at a time.

aws --profile $awsProfile s3 cp "$s3path" "$tofile" &
pids[${npids}]=$! ## save the spawned pid
let "npids=npids+1"

followed by

echo "waiting on $npids downloads"
for pid in ${pids[*]}; do
    echo $pid
    wait $pid
done

I downloaded 1500+ files (72,000 bytes) in about a minute

2

I think the simplest way to download or upload files is to use the aws s3 sync command. You can also use it to sync two S3 buckets at the same time.

aws s3 sync <LocalPath> <S3Uri> or <S3Uri> <LocalPath> or <S3Uri> <S3Uri>

# Download file(s)
aws s3 sync s3://<bucket_name>/<file_or_directory_path> .

# Upload file(s)
aws s3 sync . s3://<bucket_name>/<file_or_directory_path>

# Sync two buckets
aws s3 sync s3://<1st_s3_path> s3://<2nd_s3_path>
1
  • 1
    This is perfect. Download and upload both.
    – Nitb
    Commented Sep 9 at 7:47
1

Also you could use the --include "filename" many times in a single command with each time including a different filename within the double quotes, e.g.

aws s3 mycommand --include "file1" --include "file2"

It will save your time rather than repeating the command to download one file at a time.

1
  • for me this really slows down the sync process. I think it will iterate over all files for each include. This takes a lot of time. Maybe the only way to solve this is multithreadding, since aws cli is doing the comaprisons locally.
    – Ben
    Commented Apr 29, 2020 at 12:08
1

Also if you are running Windows(tm), WinSCP now allows drag and drop of a selection of multiple files. Including sub-folders.

Many enterprise workstations will have WinSCP installed for editing files on servers by means of SSH.

I am not affiliated, I simply think this was really worth doing.

1

In my case Aur's didn't work and if you're looking for a quick solution to download all files in a folder just using the browser, you can try entering this snippet in your dev console:

(function() {
    const rows = Array.from(document.querySelectorAll('.fix-width-table tbody tr'));
    const downloadButton = document.querySelector('[data-e2e-id="button-download"]');
    const timeBetweenClicks = 500;

    function downloadFiles(remaining) {
        if (!remaining.length) {
            return
        }

        const row = remaining[0];
        row.click();
        downloadButton.click();

        setTimeout(() => {
            downloadFiles(remaining.slice(1));
        }, timeBetweenClicks)
    }

    downloadFiles(rows)
}())
0
1

I have done, by creating shell script using aws cli (i.e : example.sh)

#!/bin/bash
aws s3 cp s3://s3-bucket-path/example1.pdf LocalPath/Download/example1.pdf
aws s3 cp s3://s3-bucket-path/example2.pdf LocalPath/Download/example2.pdf

give executable rights to example.sh (i.e sudo chmod 777 example.sh)

then run your shell script ./example.sh

0

What I usually do is mount the s3 bucket (with s3fs) in a linux machine and zip the files I need into one, then I just download that file from any pc/browser.

# mount bucket in file system
/usr/bin/s3fs s3-bucket -o use_cache=/tmp -o allow_other -o uid=1000 -o mp_umask=002 -o multireq_max=5 /mnt/local-s3-bucket-mount

# zip files into one
cd /mnt/local-s3-bucket-mount
zip all-processed-files.zip *.jpg
0

I use CloudShell to download a folder in S3

aws s3 ls
aws s3 sync s3://my-bucket/documents aaa
zip -r z.zip aaa
pwd

Then in the Download dialog

/home/cloudshell-user/z.zip
-3
import os
import boto3
import json

s3 = boto3.resource('s3', aws_access_key_id="AKIAxxxxxxxxxxxxJWB",
                    aws_secret_access_key="LV0+vsaxxxxxxxxxxxxxxxxxxxxxry0/LjxZkN")
my_bucket = s3.Bucket('s3testing')

# download file into current directory
for s3_object in my_bucket.objects.all():
    # Need to split s3_object.key into path and file name, else it will give error file not found.
    path, filename = os.path.split(s3_object.key)
    my_bucket.download_file(s3_object.key, filename)
1
  • this doesnt download "multiple" files at once
    – Anton
    Commented Jun 6, 2021 at 22:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.