I've got 200k files in a bucket which I need to move into a sub folder within the same bucket, whats the best approach?
I recently encountered the same problem. I solved it using the command line API.
http://docs.aws.amazon.com/cli/latest/index.html http://docs.aws.amazon.com/cli/latest/reference/s3/mv.html
aws s3 mv s3://BUCKETNAME/myfolder/photos/ s3://BUCKETNAME/myotherfolder/photos/ --recursive
I had a need for the objects to be publicly viewable, so I added the --acl public-read
option.
-
1IIRC, you will be charged for each command copy and delete that the mv will make in the background, as AWS do not have a "move" command, aws script will fakes that using the copy and then delete for each file recursively – higuita Jan 26 '16 at 17:46
-
@higuita As per my understanding, there is no way out of this. Move/Rename is not provided by AWS. – saurabheights Sep 20 '16 at 17:19
-
1This saved my day for moving 700,000 files to another directory! – physicalattraction Aug 25 '17 at 9:04
-
4Copy/delete is no problem, since delete requests are free. PUT, COPY, POST, or LIST cost $0.005 per 1,000 requests (standard). So e.g. moving 700,000 objects is $3.50 (same as only copying them). – jox Nov 26 '17 at 11:52
-
1I've edited this answer for security reasons. Putting
--acl public-read
in the main code body is problematic for people that don't know AWS or its security model and tend to rapidly copy/paste and try code without understanding it (many, many people). It also doesn't answer the question which doesn't mention public ACL so I put that option code at the end with the comment. – rjurney Nov 12 '20 at 17:22
Recently was able to do this with one command. Went much faster than individual requests for each file too.
Running a snippet like this:
aws s3 mv s3://bucket-name/ s3://bucket-name/subfolder --recursive --exclude "*" --include "*.txt"
Use the --include
flag to selectively pick up the files you want
-
5
-
3A word of warning: This can cause objects to be made in your bucket with a key composed of 'deeply-nested' subfolders, e.g. s3://bucket-name/subfolder/subfolder/subfolder/blah.txt. Circumvent this problem by adding an additional exclude: $ aws s3 mv s3://bucket-name/ s3://bucket-name/subfolder --recursive --exclude "" --include ".txt" --exclude "subfolder/*" – Calleniah Jun 25 '20 at 15:00
There is no 'Rename' operation though it would be great if there was.
Instead, you need to loop through each item that you want to rename, perform a copy to a new object and then a delete on the old object.
- http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectCOPY.html
- http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectDELETE.html
Note: for simplistic purposes I'm assuming you don't have versioning enabled on your bucket.
-
2that sounds very painful! API seems to be optimized for making money, not for being convenient. – Tilo Aug 2 '13 at 1:18
I had this same problem and I ended up using aws s3 mv
along with a bash for
loop.
I did aws ls bucket_name
to get all of the files in the bucket. Then I decided which files I wanted to move and added them to file_names.txt
.
Then I ran the following snippet to move all of the files:
for f in $(cat file_names.txt)
do
aws s3 mv s3://bucket-name/$f s3://bucket-name/subfolder/$f
done
if your files are in a folder, you can use s3cmd tool
s3cmd cp --recursive s3://bucket/folder/ s3://bucket/sub_folder/
Ps: I'm assuming you have already installed and configured s3cmd
The below script works perfectly to me without any issues
for i in `cat s3folders`
do
aws s3 mv s3://Bucket_Name/"$i"/ s3://Another_Bucket_Name/ --recursive
done
It also delete the empty folder from source once the files moved to the target.
-
How does Another_Bucket_Name meet the "same bucket" requirement? Where does s3folders come from? – Stevko Jun 11 '20 at 8:09