161

I have the following folder structure in S3. Is there a way to recursively remove all files under a certain folder (say foo/bar1 or foo or foo/bar2/1 ..)

foo/bar1/1/..
foo/bar1/2/..
foo/bar1/3/..

foo/bar2/1/..
foo/bar2/2/..
foo/bar2/3/..

14 Answers 14

270

With the latest aws-cli python command line tools, to recursively delete all the files under a folder in a bucket is just:

aws s3 rm --recursive s3://your_bucket_name/foo/

Or delete everything under the bucket:

aws s3 rm --recursive s3://your_bucket_name

If what you want is to actually delete the bucket, there is one-step shortcut:

aws s3 rb --force s3://your_bucket_name

which will remove the contents in that bucket recursively then delete the bucket.

Note: the s3:// protocol prefix is required for these commands to work

17
  • 2
    this should be the answer. It's a (new-ish) standard, powerful tool, designed for things just like this question Mar 4, 2015 at 21:17
  • 1
    @Naveen as I said above, rm will only delete files but rb --force will delete the files and the bucket.
    – number5
    Jun 8, 2015 at 10:53
  • 9
    using --recursive deletes the folder as well.
    – ryantuck
    Sep 29, 2016 at 22:12
  • 2
    @RyanTuck do you know how to stop it from removing the folder?
    – Moseleyi
    Mar 10, 2017 at 15:40
  • 4
    @Moseleyi i believe that you can't actually have an empty folder in an s3 bucket
    – ryantuck
    Mar 12, 2017 at 22:07
65

This used to require a dedicated API call per key (file), but has been greatly simplified due to the introduction of Amazon S3 - Multi-Object Delete in December 2011:

Amazon S3's new Multi-Object Delete gives you the ability to delete up to 1000 objects from an S3 bucket with a single request.

See my answer to the related question delete from S3 using api php using wildcard for more on this and respective examples in PHP (the AWS SDK for PHP supports this since version 1.4.8).

Most AWS client libraries have meanwhile introduced dedicated support for this functionality one way or another, e.g.:

Python

You can achieve this with the excellent boto Python interface to AWS roughly as follows (untested, from the top of my head):

import boto
s3 = boto.connect_s3()
bucket = s3.get_bucket("bucketname")
bucketListResultSet = bucket.list(prefix="foo/bar")
result = bucket.delete_keys([key.name for key in bucketListResultSet])

Ruby

This is available since version 1.24 of the AWS SDK for Ruby and the release notes provide an example as well:

bucket = AWS::S3.new.buckets['mybucket']

# delete a list of objects by keys, objects are deleted in batches of 1k per
# request.  Accepts strings, AWS::S3::S3Object, AWS::S3::ObectVersion and 
# hashes with :key and :version_id
bucket.objects.delete('key1', 'key2', 'key3', ...)

# delete all of the objects in a bucket (optionally with a common prefix as shown)
bucket.objects.with_prefix('2009/').delete_all

# conditional delete, loads and deletes objects in batches of 1k, only
# deleting those that return true from the block
bucket.objects.delete_if{|object| object.key =~ /\.pdf$/ }

# empty the bucket and then delete the bucket, objects are deleted in batches of 1k
bucket.delete!

Or:

AWS::S3::Bucket.delete('your_bucket', :force => true)
1
50

You might also consider using Amazon S3 Lifecycle to create an expiration for files with the prefix foo/bar1.

Open the S3 browser console and click a bucket. Then click Properties and then LifeCycle.

Create an expiration rule for all files with the prefix foo/bar1 and set the date to 1 day since file was created.

Save and all matching files will be gone within 24 hours.

Just don't forget to remove the rule after you're done!

No API calls, no third party libraries, apps or scripts.

I just deleted several million files this way.

A screenshot showing the Lifecycle Rule window (note in this shot the Prefix has been left blank, affecting all keys in the bucket):

enter image description here

4
  • 4
    Great idea for using Lifecycle instead of some delete command.
    – xis
    Jun 4, 2014 at 17:11
  • Exactly, let S3 do it for you.
    – Ryan
    Jun 25, 2014 at 21:29
  • You can also apply this to the entire bucket, enabling you to delete the bucket.
    – Indolering
    Nov 22, 2014 at 23:17
  • I wonder if you could use "0" as the Time Period for days expiration, to queue it's immediate delete?
    – Brad Parks
    Jun 1 at 18:10
20

The voted up answer is missing a step.

Per aws s3 help:

Currently, there is no support for the use of UNIX style wildcards in a command's path arguments. However, most commands have --exclude "<value>" and --include "<value>" parameters that can achieve the desired result......... When there are multiple filters, the rule is the filters that appear later in the command take precedence over filters that appear earlier in the command. For example, if the filter parameters passed to the command were --exclude "*" --include "*.txt" All files will be excluded from the command except for files ending with .txt

aws s3 rm --recursive s3://bucket/ --exclude="*" --include="/folder_path/*" 
0
12

With s3cmd package installed on a Linux machine, you can do this

s3cmd rm s3://foo/bar --recursive

2
  • 2
    According to the help it is either single-object delete s3cmd del s3://BUCKET/OBJECT or whole bucket delete s3cmd rb s3://BUCKET. There is no s3cmd rm, at least according to s3cmd --help. Sep 24, 2015 at 0:30
  • 2
    s3cmd rm is in the help as of 2019 (as an alias for del), this is an excellent answer. The aws cli tools only work against a / terminating prefix, but not a folder and partial filename prefix, whereas s3cmd works in both cases. This answer needs lots more upvotes, I had to scroll way too far to find the right solution. Sep 20, 2019 at 18:20
9

In case if you want to remove all objects with "foo/" prefix using Java AWS SDK 2.0

import java.util.ArrayList;
import java.util.Iterator;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.*;

//...

ListObjectsRequest listObjectsRequest = ListObjectsRequest.builder()
    .bucket(bucketName)
    .prefix("foo/")
    .build()
;
ListObjectsResponse objectsResponse = s3Client.listObjects(listObjectsRequest);

while (true) {
    ArrayList<ObjectIdentifier> objects = new ArrayList<>();

    for (Iterator<?> iterator = objectsResponse.contents().iterator(); iterator.hasNext(); ) {
        S3Object s3Object = (S3Object)iterator.next();
        objects.add(
            ObjectIdentifier.builder()
                .key(s3Object.key())
                .build()
        );
    }

    s3Client.deleteObjects(
        DeleteObjectsRequest.builder()
            .bucket(bucketName)
            .delete(
                Delete.builder()
                    .objects(objects)
                    .build()
            )
            .build()
    );

    if (objectsResponse.isTruncated()) {
        objectsResponse = s3Client.listObjects(listObjectsRequest);
        continue;
    }

    break;
};
2
  • 2
    I can't find a more telling demonstration of what people don't like about Java than this answer...
    – Jivan
    May 22, 2020 at 17:45
  • @Jivan Java is not the problem. It is the s3 client library. Feb 5 at 15:51
4

In case using AWS-SKD for ruby V2.

s3.list_objects(bucket: bucket_name, prefix: "foo/").contents.each do |obj|
  next if obj.key == "foo/" 
  resp = s3.delete_object({
    bucket: bucket_name,
    key: obj.key,
  })
end

attention please, all "foo/*" under bucket will delete.

4

To delete all the versions of the objects under a specific folder:

Pass the path /folder/subfolder/ to the Prefix -


import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket("my-bucket-name")
bucket.object_versions.filter(Prefix="foo/bar1/1/").delete()
3

I just removed all files from my bucket by using PowerShell:

Get-S3Object -BucketName YOUR_BUCKET | % { Remove-S3Object -BucketName YOUR_BUCKET -Key $_.Key -Force:$true }
1
  • Thanks for posting this answer, I was trying to do this exact thing and had put -Key "%_.Key" which doesn't work. Jan 13, 2015 at 19:20
3

Just saw that Amazon added a "How to Empty a Bucket" option to the AWS console menu:

http://docs.aws.amazon.com/AmazonS3/latest/UG/DeletingaBucket.html

2

Best way is to use lifecycle rule to delete whole bucket contents. Programmatically you can use following code (PHP) to PUT lifecycle rule.

$expiration = array('Date' => date('U', strtotime('GMT midnight')));
$result = $s3->putBucketLifecycle(array(
            'Bucket' => 'bucket-name',
            'Rules' => array(
                array(
                    'Expiration' => $expiration,
                    'ID' => 'rule-name',
                    'Prefix' => '',
                    'Status' => 'Enabled',
                ),
            ),
        ));

In above case all the objects will be deleted starting Date - "Today GMT midnight".

You can also specify Days as follows. But with Days it will wait for at least 24 hrs (1 day is minimum) to start deleting the bucket contents.

$expiration = array('Days' => 1);
1

I needed to do the following...

def delete_bucket
  s3 = init_amazon_s3
  s3.buckets['BUCKET-NAME'].objects.each do |obj|
    obj.delete
  end
end

def init_amazon_s3
  config = YAML.load_file("#{Rails.root}/config/s3.yml")
  AWS.config(:access_key_id => config['access_key_id'],:secret_access_key => config['secret_access_key'])
  s3 = AWS::S3.new
end
1
s3cmd del --recursive s3://your_bucket --force
0

In case you want to use aws javascript sdk v3, You can do it using following commands:

  • ListObjectsV2Command
  • DeleteObjectsCommand

Following is the javascript code that you can execute in NnodeJS runtime to remove all files of specific folder:

const { S3Client, ListObjectsV2Command, DeleteObjectsCommand } = require("@aws-sdk/client-s3");

// specify your region 
const AWS_REGION = 'ap-south-1';

// init s3 client
const s3Client = new S3Client({ region: AWS_REGION });

/**
 * S3 utility class
 */
class S3Utility {

  /**
   * private function to send delete request to s3
   * 
   * @param s3Bucket 
   * @param s3Objects 
   * 
   * @returns boolean
   */
  async #deleteObjectsRequest(s3Bucket, s3Objects) {
    try {
      const objects = [];
      s3Objects.forEach((object) => {
        if (object.Key) {
          objects.push({ Key: object.Key });
        }
      });

      const deleteResponse = await s3Client.send(new DeleteObjectsCommand({
        Bucket: s3Bucket,
        Delete: {
          Objects: objects
        }
      }));

      return deleteResponse['$metadata'];
    } catch (error) {
      console.log('S3Utility.#deleteObjectsRequest : error ', error);
      return false;
    }
  }

  /**
   * Delete all folder and files of s3 folder specified by s3BucketName and s3Folder
   * 
   * @param s3BucketName S3 bucket name
   * @param s3Folder S3 folder path, e.g data/site/images
   * 
   * @returns boolean
   */
  async deleteS3Folder(s3BucketName, s3Folder) {
    try {
      const s3Response = await s3Client.send(new ListObjectsV2Command({ Bucket: s3BucketName, Prefix: s3Folder }));
      const objectExists = (s3Response['$metadata']['httpStatusCode'] === 200
        && s3Response.hasOwnProperty('Contents')
        && Array.isArray(s3Response.Contents)
        && s3Response.Contents.length > 0) ? true : false;

      if (objectExists) {
        await this.#deleteObjectsRequest(s3BucketName, s3Response.Contents);
        await this.deleteS3Folder(s3BucketName, s3Folder);
      }

      return true;
    } catch (error) {
      console.log('S3Utility.#deleteS3Folder : error ', error);
      return false;
    }

  }

}

// use our s3 utility class using async/await 
async () => {
  try {
      const s3Utility = new S3Utility();
      await s3Utility.deleteS3Folder('my-bucket-name', 'data/site/images');
  } catch (err) {
      console.log(err);
  }
}

More references:

1
  • why do you need to call await this.deletes3folder(...) again recursively? Jul 13 at 23:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.