17

I have an Amazon S3 bucket with versioning enabled. Due to a misconfigured lifecycle policy, many of the objects in this bucket had Delete Markers added to them.

I can remove these markers from the S3 console to restore the previous versions of these objects, but there are enough objects to make doing this manually on the web console extremely time-inefficient.

Is there a way to find all Delete Markers in an S3 bucket and remove them, restoring all files in that bucket? Ideally I would like to do this from the console itself, although I will happily write a script or use the amazon CLI tools to do this if that's the only way.

Thanks!

10 Answers 10

25

Use this to restore the files inside the specific folder. I've used aws cli commands in my script. Provide input as: sh scriptname.sh bucketname path/to/a/folder

**Script:**
#!/bin/bash
#please provide the bucketname and path to destination folder to restore
# Remove all versions and delete markers for each object
 aws s3api list-object-versions --bucket $1 --prefix $2 --output text | 
 grep "DELETEMARKERS" | while read obj
   do
        KEY=$( echo $obj| awk '{print $3}')
        VERSION_ID=$( echo $obj | awk '{print $5}')
        echo $KEY
        echo $VERSION_ID
        aws s3api delete-object --bucket $1 --key $KEY --version-id $VERSION_ID

   done

Edit: put $VERSION_ID in correct position in the script

5
  • I like that, as it can be easily combined with other filters (e.g. grep -P pattern).
    – rudi
    Dec 13, 2017 at 15:54
  • 2
    This doesn't work with folders that have spaces in their names Oct 9, 2018 at 19:30
  • Plesae replace the version Id part with the following to make this work on special charecter naming objects: VERSION_ID=$(aws s3api list-object-versions --bucket $1 --prefix "$obj" --query DeleteMarkers[*].VersionId --output text)
    – Kc Bickey
    Dec 27, 2018 at 3:26
  • 1
    Watch out! If you have spaces in your filename, this doesn't work... Since awk is looking for spaces
    – Daan
    Jul 24, 2020 at 16:02
  • Thank you for this - really great! I ran into the file spaces issues, tried to add -F "\t" to the awk but that didn't work because once it is read into the obj variable the tab is gone. So I ended making temporary files - 1 for output to find the delete marker, another to extract columns 3 and 5 using awk with tab delimiter -F "\t" and formating the output with "--9283dll8338-- as the separator (made up string that shouldn't occur anywhere else)
    – dllahr
    Jun 26 at 8:41
21

Here's a sample Python implementation:

import boto3
import botocore

BUCKET_NAME = 'BUCKET_NAME'
s3 = boto3.resource('s3')


def main():
    bucket = s3.Bucket(BUCKET_NAME)
    versions = bucket.object_versions

    for version in versions.all():
        if is_delete_marker(version):
             version.delete()


def is_delete_marker(version):
    try:
        # note head() is faster than get()
        version.head()
        return False
    except botocore.exceptions.ClientError as e:
        if 'x-amz-delete-marker' in e.response['ResponseMetadata']['HTTPHeaders']:
            return True
        # an older version of the key but not a DeleteMarker
        elif '404' == e.response['Error']['Code']:
            return False


if __name__ == '__main__':
    main()

For some context for this answer see: https://docs.aws.amazon.com/AmazonS3/latest/dev/DeleteMarker.html

If you try to get an object and its current version is a delete marker, Amazon S3 responds with:

  • A 404 (Object not found) error
  • A response header, x-amz-delete-marker: true

The response header tells you that the object accessed was a delete marker. This response header never returns false; if the value is false, Amazon S3 does not include this response header in the response.

The only way to list delete markers (and other versions of an object) is by using the versions subresource in a GET Bucket versions request. A simple GET does not retrieve delete marker objects.

Unfortunately, despite what is written in https://github.com/boto/botocore/issues/674, checking if ObjectVersion.size is None is not a reliable way to determine if a version is a delete marker as it will also be true for previously deleted versions of folder keys.

Currently, boto3 is missing a straightforward way to determine if an ObjectVersion is a DeleteMarker. See https://github.com/boto/boto3/issues/1769

However, ObjectVersion.head() and .Get() operations will throw an exception on an ObjectVersion that is a DeleteMarker. Catching this exception is likely the only reliable way of determining if an ObjectVersion is a DeleteMarker.

1
  • In case someone would like to have an example of filtering on bucket prefixes: <br/> <br/> key = 'bucket_prefix1/bucket_prefix2' for version in versions.all(): if key in version.object_key: if is_delete_marker(version): version.delete()
    – rbigley
    Feb 8 at 20:23
10

Define variables

PROFILE="personal"
REGION="eu-west-1"
BUCKET="mysql-backend-backups-prod"

Delete DeleteMarkers at once

aws --profile $PROFILE s3api delete-objects \
    --region $REGION \
    --bucket $BUCKET \
    --delete "$(aws --profile $PROFILE s3api list-object-versions \
                    --region $REGION \
                    --bucket $BUCKET \
                    --output=json \
                    --query='{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}')"

Delete versions at once

aws --profile $PROFILE s3api delete-objects \
    --region $REGION \
    --bucket $BUCKET \
    --delete "$(aws --profile $PROFILE s3api list-object-versions \
                    --region $REGION \
                    --bucket $BUCKET \
                    --output=json \
                    --query='{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"

And delete S3 bucket afterward

aws --profile $PROFILE s3api delete-bucket \
    --region $REGION \
    --bucket $BUCKET
1
  • 3
    This worked great for me! Note that a --prefix might be needed in the event that the user only wants to delete certain objects, but not all. (the queries without --prefix tend to be pretty slow on reasonably-sized buckets...) Jan 14, 2019 at 22:07
9

I just wrote a program (using boto) to solve the same problem:

from boto.s3 import deletemarker
from boto.s3.connection import S3Connection
from boto.s3.key import Key

def restore_bucket(bucket_name): 
    bucket = conn.get_bucket(bucket_name)
    for version in bucket.list_versions():
        if isinstance(version, deletemarker.DeleteMarker) and version.is_latest:
            bucket.delete_key(version.name, version_id=version.version_id)

If you need to restore folders within the versioned buckets, the rest of the program I wrote can be found here.

0
5

You would need to write a program to:

  • Loop through all objects in the Amazon S3 bucket
  • Retrieve the version IDs for each version of each object
  • Delete the delete markers

This could be done fairly easily using the SDK, such as boto.

The AWS Command-Line Interface (CLI) can also be used, but you would have to build a script around it to capture the IDs and then delete the markers.

1
  • 1
    Thanks, this pointed me in the right direction. Ultimately, I grabbed all delete markers with ListObjectVersions, then deleted them with the API
    – Dave Stern
    Aug 20, 2015 at 18:31
4

Most of the above versions are very slow on large buckets as they use delete-object rather than delete-objects. Here a variant on the bash version which uses awk to issue 100 requests at a time:

Edit: just saw @Viacheslav's version which also uses delete-objects and is nice and clean, but will fail with large numbers of markers due to line length issues.

#!/bin/bash

bucket=$1
prefix=$2

aws s3api list-object-versions \
    --bucket "$bucket" \
    --prefix "$prefix" \
    --query 'DeleteMarkers[][Key,VersionId]' \
    --output text |
awk '{ acc = acc "{Key=" $1 ",VersionId=" $2 "}," }
     NR % 100 == 0 {print "Objects=[" acc "],Quiet=False"; acc="" }
     END { print "Objects=[" acc "],Quiet=False" }' |
while read batch; do
    aws s3api delete-objects --bucket "$bucket" --delete "$batch" --output text
done
2
  • This doesn't work for prefixes and objects with spaces in their names. Can you modify this to cater for that? Feb 25, 2021 at 20:02
  • @patricknwakwoke you can use the following adjustments change this awk '{ acc = acc "{Key=" $1 ",VersionId=" $2 "}," } to awk -F"\t" '{ acc = acc "{Key=" $1 ",VersionId=" $2 "}," } and you would run into another set of issues when Key=None so add this before while loop: awk '{gsub(/{Key=None,VersionId=},/, "");}1' | Apr 21, 2021 at 22:17
2

I have been dealing with this problem a few weeks ago.

Finally I managed to generate a function in PHP that deletes the 'deleted markers' of the latest version of the files within a prefix. Personally, it worked perfectly and, in a pass of this script, iterating through all the prefixes, I managed to mend my own error by deleting many s3 objects unintentionally.

I leave my implementation in PHP below :

private function restore_files($file)
{
    $storage = get_storage()->getDriver()->getAdapter()->getClient();
    $bucket_name = 'my_bucket_name';
    $s3_path=$file->s3_path;

    $restore_folder_path = pathinfo($s3_path, PATHINFO_DIRNAME);

    $data = $storage->listObjectVersions([
        'Bucket' => $bucket_name,
        'Prefix' => $restore_folder_path,
    ]);

    $data_array = $data->toArray();
    $deleteMarkers = $data_array['DeleteMarkers'];

    foreach ($deleteMarkers as $key => $delete_marker) {
        if ($delete_marker["IsLatest"]) {
            $objkey = $delete_marker["Key"];
            $objVersionId = $delete_marker["VersionId"];

            $delete_response = $storage-> deleteObjectAsync([
                'Bucket' => $bucket_name,
                'Key' => $objkey,
                'VersionId' => $objVersionId
            ]);
        }
    }
}

Some considerations about the script:

  1. The code was implemented using Laravel Framework so, in the variable $storage, i get the PHP SDK alone, without using all the laravel's wrapper. So, $storage varible is the Client Object of the S3 SDK. Here is the documentation that I have used.
  2. The $file parameter that the function receives, is an objected that has the s3_path in their's properties. So, in the $restore_folder_path varible, I get the prefix of the object s3 path.
  3. Finally, i get all the objects inside the prefix in s3. I iterate over the DeleteMarkers list, and ask if the current object is the lasted deleted marker. If it is, i make a post to deleteObject function with the specif id of the object that i want to delete it's deleted marker. This is the way s3 documentation specify to remove the deleted marker
0
1

Set up a life cycle rule to remove them after a certain days. Otherwise it will cost you 0.005$ per 1000 Object Listing.

So most efficient way is setting up a lifecycle rule.

Here is the step by step method. https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html

1
  • this only allows expired delete marker expiration how do I delete all delete markers? Jan 14, 2021 at 15:42
0
I checked the file size. 
Marker size is 'None'
Remove all Marker.
import boto3

default_session=boto3.session.Session(profile_name="default")
s3_re=default_session.resource(service_name="s3", region_name="ap-northeast-2")
for each_bucket in s3_re.buckets.all():
    bucket_name = each_bucket.name
    s3 = boto3.resource('s3')
    bucket = s3.Bucket(bucket_name)
    version = bucket.object_versions
    for ver in version.all():
        if str(ver.size) in 'None':
            delete_file = ver.delete()
            print(delete_file)
        else:
            pass

4
  • How to limit it to a single bucket? Jan 14, 2021 at 15:42
  • How to delete everything from s3 bucket object+objectversions+delete_markers Jan 14, 2021 at 21:23
  • Your code doesn't work as you are setting session twice Jan 14, 2021 at 21:32
  • could you explain what your code is doing? Jan 14, 2021 at 21:58
0

This code works with or without prefixes but if u use it without prefix I am not pretty sure it will deep dig enough for the 15 minutes timeout. I created a AWS lambda function using Node.js 14 (put the timeout up to 15minutes) and I ran it manually (it can be automated) but I had like 10 folders in one of the applications bucket that had been deleted and since I had versioning turned on there were versions with Delete Marker. So I had to loop through all the pages and find the ones with delete marker and restore them with their last version so they can be used again. The biggest issue was that while thinking through this was that I used recursion but I get the same 1000 objects since this is how AWS list objects function works and in order to get the next 1000 there is a property called NextKeyMarker that did the job for my case. And the flag that indicates if there are more pages is : IsTruncated - if it is true means that there are more pages - so invoke that function again with the nextKeyMarker to know where to start from (recursion).

const AWS = require('aws-sdk');
const s3 = new AWS.S3();

async function restoreDeletedObjects(bucketName, prefix, nextMarker) {
  try {
    const params = {
      Bucket: bucketName,
      Prefix: prefix,
      KeyMarker: nextMarker,
    };

    const response = await s3.listObjectVersions(params).promise();
    console.log(response, 'response');

    if (response.DeleteMarkers.length > 0) {
      console.log('in the first if', response.DeleteMarkers.length);
      const objectsToDelete = response.DeleteMarkers.map(deleteMarker => ({
        Key: deleteMarker.Key,
        VersionId: deleteMarker.VersionId,
      }));

      console.log(objectsToDelete, 'objects to delete');
      // Remove the delete markers and restore the objects
      await s3.deleteObjects({
        Bucket: bucketName,
        Delete: { Objects: objectsToDelete, Quiet: false },
      }).promise();

      for (const deleteMarker of response.DeleteMarkers) {
        console.log(deleteMarker, 'delete Marker');
        if (deleteMarker.IsLatest) {
          console.log(`Restored: s3://${bucketName}/${deleteMarker.Key}, VersionId: ${deleteMarker.VersionId}`);
        }
      }
    }

    if (response.IsTruncated) {
      // If there are more delete markers, call the function recursively with the NextKeyMarker
      await restoreDeletedObjects(bucketName, prefix, response.NextKeyMarker);
    } else {
      console.log('Everything with Delete Marker was restored sucessfully')
    }
  } catch (err) {
    console.error("Error:", err);
  }
}

exports.handler = async (event, context) => {
  const bucketName = "bucket-name-here"; // Replace with your bucket name
  const prefix = "put-your-prefix-here"; // Replace with your desired prefix
  // we had structure like this - Buckets -> "data" -> and 10 folders here. so we used like it that for each folder that had more than 15000 objects  -> 
  // const bucketName = "orders"
  // const prefix = "data/folder1" then "data/folder2" and we ran that for each folder.

  await restoreDeletedObjects(bucketName, prefix, undefined);
};```

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.