I have to upload 100k small files (Total size: 200MB).
I've tried to do that via web browser (AWS Console), but for the first 15 min, i've uploaded only 2MB.
What is the fastest way to upload 100k small files to S3?
I have 99,561 small files totalling 466MB and experimented with uploading these to S3 as quickly as possible from my m4.16xlarge
(64 CPU) EC2 instance.
aws s3 sync
took 10 minutess3-parallel-put --put=stupid
took 9 minutess3-parallel-put --put=stupid --processes=64
took 1 minutes3-parallel-put --put=stupid --processes=256
took 19 secondss3-parallel-put --put=stupid --processes=512
took 18 secondsI'll be moving forward with a s3-parallel-put
solution.
./s3-parallel-put --bucket=ililililili --bucket_region=nyc3.digitaloceanspaces.com --grant=public-read --host=nyc3.digitaloceanspaces.com --prefix=test/upload/ --put=stupid --processes=30 --content-type=guess --log-filename=./s3pp.log ./uploads/
Jul 13, 2021 at 16:33
Install the AWS CLI tool, then use the aws s3 cp
command or the aws s3 sync
command to upload all your files to S3.
Alternatively you could look into using third-party S3 clients such as Cyberduck and CloudBerry Explorer.