23

I have a versioned bucket and would like to delete the object (and all of its versions) from the bucket. However, when I try to delete the object from the console, S3 simply adds a delete marker but does not perform a hard delete.

Is it possible to delete all versions of the object (hard delete) with a particular key?:

s3resource = boto3.resource('s3')
bucket = s3resource.Bucket('my_bucket')
obj = bucket.Object('my_object_key')

# I would like to delete all versions for the object like so:
obj.delete_all_versions()

# or delete all versions for all objects like so:
bucket.objects.delete_all_versions()
9
  • delete_objects should let you delete objects regardless of the version. I'm not 100% sure
    – Mangohero1
    Oct 18, 2017 at 21:35
  • 1
    ok, I will have to test this. It has an optional VersionId parameter, so that makes me think that if I do not explicitly provide the version id for each object, it will just perform a soft delete (delete marker only).
    – rooscous
    Oct 18, 2017 at 21:49
  • Nope.. just tried it. thought it would just delete all of them if I didn't specify the VersionId parameter, but just did the latest one
    – Mangohero1
    Oct 18, 2017 at 21:50
  • If I have to provide each version id explicitly, then I will have to get the list of version ids and iterate over each one individually. Really don't want to have to do that.
    – rooscous
    Oct 18, 2017 at 21:51
  • 1
    I highly recommend the documentation: docs.aws.amazon.com/AmazonS3/latest/dev/….
    – jarmod
    Oct 18, 2017 at 21:57

12 Answers 12

18

The other answers delete objects individually. It is more efficient to use the delete_objects boto3 call and batch process your delete. See the code below for a function which collects all objects and deletes in batches of 1000:

bucket = 'bucket-name'
s3_client = boto3.client('s3')
object_response_paginator = s3_client.get_paginator('list_object_versions')

delete_marker_list = []
version_list = []

for object_response_itr in object_response_paginator.paginate(Bucket=bucket):
    if 'DeleteMarkers' in object_response_itr:
        for delete_marker in object_response_itr['DeleteMarkers']:
            delete_marker_list.append({'Key': delete_marker['Key'], 'VersionId': delete_marker['VersionId']})

    if 'Versions' in object_response_itr:
        for version in object_response_itr['Versions']:
            version_list.append({'Key': version['Key'], 'VersionId': version['VersionId']})

for i in range(0, len(delete_marker_list), 1000):
    response = s3_client.delete_objects(
        Bucket=bucket,
        Delete={
            'Objects': delete_marker_list[i:i+1000],
            'Quiet': True
        }
    )
    print(response)

for i in range(0, len(version_list), 1000):
    response = s3_client.delete_objects(
        Bucket=bucket,
        Delete={
            'Objects': version_list[i:i+1000],
            'Quiet': True
        }
    )
    print(response)
5
  • 3
    Excellent, I didn't know about delete_objects. Out of curiosity, why do you build the two lists, delete_marker_list and version_list independently and then iterate over them independently? Would it work to just build one list combining all versions and delete markers, then iterate over that single list in steps of 1000?
    – gene_wood
    Jan 8, 2019 at 6:15
  • 4
    Just for clarity.
    – AndrewC
    Jan 8, 2019 at 20:33
  • 1
    Thanks a lot! Used it to nuke all not latest versions (not version['IsLatest']) after disabling bucket versioning. Feb 12, 2020 at 19:41
  • This is really great example . Thank you
    – Emma Y
    Apr 3, 2020 at 23:00
  • Evgeny makes a great point- this deletes all versions, including the current one, unless you specify otherwise! May 18, 2023 at 14:42
10

I had trouble using the other solutions to this question so here's mine.

import boto3
bucket = "bucket name goes here"
filename = "filename goes here"

client = boto3.client('s3')
paginator = client.get_paginator('list_object_versions')
response_iterator = paginator.paginate(Bucket=bucket)
for response in response_iterator:
    versions = response.get('Versions', [])
    versions.extend(response.get('DeleteMarkers', []))
    for version_id in [x['VersionId'] for x in versions
                       if x['Key'] == filename and x['VersionId'] != 'null']:
        print('Deleting {} version {}'.format(filename, version_id))
        client.delete_object(Bucket=bucket, Key=filename, VersionId=version_id)

This code deals with the cases where

  • object versioning isn't actually turned on
  • there are DeleteMarkers
  • there are no DeleteMarkers
  • there are more versions of a given file than fit in a single API response

Mahesh Mogal's answer doesn't delete DeleteMarkers. Mangohero1's answer fails if the object is missing a DeleteMarker. Hari's answer repeats 10 times (to workaround missing pagination logic).

6
  • Can you extend it to delete all files and all delete markers, please? Basically a version that will, when completed, allow deleting a bucket that will be empty. Apr 23, 2020 at 11:42
  • @IgorKryltsov It should delete DeleteMarkers. The line that says ` versions.extend(response.get('DeleteMarkers', []))` is meant to iterate over DeleteMarkers as well and delete them. When you run it does it not do that?
    – gene_wood
    Apr 24, 2020 at 14:20
  • It is probably outside of the scope of this question, sorry. I want to empty versioned bucket, including delete markers, so I can drop this bucket. Apr 26, 2020 at 4:14
  • @IgorKryltsov Indeed, the code above should empty a versioned bucket, including delete markers. When you run it does it not do this?
    – gene_wood
    Apr 26, 2020 at 5:09
  • 1
    how do I use it via s3 = session.resource('s3') bucket = s3.Bucket(BUCKET) Jan 14, 2021 at 20:29
10

You can use object_versions.

def delete_all_versions(bucket_name: str, prefix: str):
    s3 = boto3.resource('s3')
    bucket = s3.Bucket(bucket_name)
    if prefix is None:
        bucket.object_versions.delete()
    else:
        bucket.object_versions.filter(Prefix=prefix).delete()

delete_all_versions("my_bucket", None) # empties the entire bucket
delete_all_versions("my_bucket", "my_prefix/") # deletes all objects matching the prefix (can be only one if only one matches)
9

The documentation is helpful here:

  1. When versioning is enabled in an S3 bucket, a simple DeleteObject request cannot permanently delete an object from that bucket. Instead, Amazon S3 inserts a delete marker (which is effectively a new version of the object with its own version ID).
  2. When you try to GET an object whose current version is a delete marker, S3 behaves as if the object has been deleted (even though it has not) and returns a 404 error.
  3. To permanently delete an object from a versioned bucket, use DeleteObject, with the relevant version ID, for each and every version of the object (and that includes the delete markers).
2
  • Hi jarmod. Thanks for the response -- I did read the documentation, but I wanted to know if this is really the only way to do it (i.e. get a list of all versions of the object and delete each one iteratively).
    – rooscous
    Oct 19, 2017 at 16:38
  • 1
    This should really be upvoted more as it sheds light on the actual workings through documentation.
    – San
    Apr 15, 2019 at 16:17
5

To delete all versions of an object or objects under a prefix:

Pass the object key /folder/filename or prefix /folder/subfolder/ to the Prefix

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket("my-bucket-name")
bucket.object_versions.filter(Prefix="folder/subfolder/").delete()
4

As a supplement to @jarmod's answer, here is a way I developed a workaround to "hard deleting" an object (with delete markered objects included);

def get_all_versions(bucket, filename):
    s3 = boto3.client('s3')
    keys = ["Versions", "DeleteMarkers"]
    results = []
    for k in keys:
        response = s3.list_object_versions(Bucket=bucket)[k]
        to_delete = [r["VersionId"] for r in response if r["Key"] == filename]
    results.extend(to_delete)
    return results

bucket = "YOUR BUCKET NAME"
file = "YOUR FILE"

for version in get_all_versions(bucket, file):
    s3.delete_object(Bucket=bucket, Key=file, VersionId=version)
1
  • 2
    If your file doesn't have a DeleteMarker this solution will throw an exception. I added a solution here that accounts for the case where a file does or doesn't have a DeleteMarker
    – gene_wood
    Dec 20, 2018 at 16:22
3

This post was super helpful without this we would have spent tremendous amount of time cleaning up our S3 folders.

We had a requirement to clean up specific folders only. So I tried the following code and it worked like a charm. Also note that I am iterating through the 10 times to delete more than 1000 objects limit that function has. Feel free to modify the limit as you wish.

import boto3
session = boto3.Session(aws_access_key_id='<YOUR ACCESS KEY>',aws_secret_access_key='<YOUR SECRET KEY>')

bucket_name = '<BUCKET NAME>'
object_name = '<KEY NAME>'

s3 = session.client('s3')

for i in range(10):
   versions = s3.list_object_versions (Bucket = bucket_name, Prefix = object_name)
#print (versions)
   version_list = versions.get('Versions')
   for version in version_list:
      keyName = version.get('Key')
      versionId = version.get('VersionId')
      print (keyName + ':' + versionId)
      s3.delete_object(Bucket = bucket_name, Key= keyName, VersionId = versionId)
   marker_list = versions.get('DeleteMarkers')
#print(marker_list)
   for marker in marker_list:
      keyName1 = marker.get('Key')
      versionId1 = marker.get('VersionId')
      print (keyName1 + ':' + versionId1)
      s3.delete_object(Bucket = bucket_name, Key= keyName1, VersionId = versionId1)
2
  • 1
    I added a solution here that uses a paginator to page through the results instead of repeating the call 10 times.
    – gene_wood
    Dec 20, 2018 at 16:23
  • @Hari can you modify answer to work with s3 = session.resource('s3') bucket = s3.Bucket(BUCKET) Jan 14, 2021 at 20:29
3

Fewer line solution.

import boto3

def delete_versions(bucket, objects=None): # `objects` is either list of str or None
  bucket = boto3.resource('s3').Bucket(bucket)
  if objects: # delete specified objects
    [version.delete() for version in bucket.object_versions.all() if version.object_key in objects]
  else: # or delete all objects in `bucket`
    [version.delete() for version in bucket.object_versions.all()]
0
2

Easiest way:

import boto3
bucket = boto3.resource("s3").Bucket("mybucket")
bucket.object_versions.all().delete()
1

this script will delete all version of all object with prefix -

s3 = boto3.resource("s3")
client = boto3.client("s3")
s3_bucket = s3.Bucket(bucket_name)
for obj in s3_bucket.objects.filter(Prefix=""):

    response = client.list_object_versions(Bucket=bucket_name, Prefix=obj.key)

    while "Versions" in response:
        to_delete = [
            {"Key": ver["Key"], "VersionId": ver["VersionId"]}
            for ver in response["Versions"]
        ]

        delete = {"Objects": to_delete}

        client.delete_objects(Bucket=bucket_name, Delete=delete)
        response = client.list_object_versions(Bucket=bucket_name, Prefix=obj.key)

    client.delete_object(Bucket=bucket_name, Key=obj.key)
0

You can delete an object with all of its versions using following code

session = boto3.Session(aws_access_key_id, aws_secret_access_key)

bucket_name = 'bucket_name'
object_name = 'object_name'

s3 = session.client('s3')

versions = s3.list_object_versions (Bucket = bucket_name, Prefix = object_name)
version_list = versions.get('Versions')
for version in version_list:
    versionId = version.get('VersionId')
    s3.delete_object(Bucket = bucket_name, Key= object_name, VersionId = versionId)
1
  • This solution doesn't delete DeleteMarkers. I added a solution here that accounts for the case where a file does or doesn't have a DeleteMarker
    – gene_wood
    Dec 20, 2018 at 16:23
0

The rest of the answers all miss something. Either using the Prefix parameter, or deleting delete markers, or handling errors...

s3 = boto3.client('s3')
response = s3.list_object_versions(Bucket=bucket_name, Prefix=key)

objects_to_delete = []
# Note that we do not use pagination because we assume the file has less than max versions (something like 300)
# Note that we also traverse delete markers.
for obj in itertools.chain(response.get("Versions", []), response.get("DeleteMarkers", [])):
    # NOTE: This is super stupid, but AWS has no API for list_object_versions for a single object, only with prefix.
    # So other objects who share the same prefix (e.g "blaze/a.txt" and "bla.json" will also be listed when asking for "bla").
    # So we need to be careful here
    if obj["Key"] != key:
        break
    objects_to_delete.append({"Key": obj["Key"], 'VersionId': obj['VersionId']})

if len(objects_to_delete) == 0:
    raise FileNotFoundError(f'File {key} not found at bucket {bucket_name}')

deletion_response = s3.delete_objects(Bucket=bucket_name, Delete={"Objects": objects_to_delete, "Quiet": False})

errors = deletion_response.get("Errors", [])
if len(errors) > 0:
    raise Exception(f'Failed deleting file {key} from bucket {bucket_name}. Result: {deletion_response}')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.