71

Is there any way to force CloudFormation to delete a non-empty S3 Bucket?

8 Answers 8

71

You can create a lambda function to clean up your bucket and invoke your lambda from your CloudFormation stack using a CustomResource.

Below a lambda example cleaning up your bucket:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import json
import boto3
from botocore.vendored import requests


def lambda_handler(event, context):
    try:
        bucket = event['ResourceProperties']['BucketName']

        if event['RequestType'] == 'Delete':
            s3 = boto3.resource('s3')
            bucket = s3.Bucket(bucket)
            for obj in bucket.objects.filter():
                s3.Object(bucket.name, obj.key).delete()

        sendResponseCfn(event, context, "SUCCESS")
    except Exception as e:
        print(e)
        sendResponseCfn(event, context, "FAILED")


def sendResponseCfn(event, context, responseStatus):
    response_body = {'Status': responseStatus,
                     'Reason': 'Log stream name: ' + context.log_stream_name,
                     'PhysicalResourceId': context.log_stream_name,
                     'StackId': event['StackId'],
                     'RequestId': event['RequestId'],
                     'LogicalResourceId': event['LogicalResourceId'],
                     'Data': json.loads("{}")}

    requests.put(event['ResponseURL'], data=json.dumps(response_body))

After you create the lambda above, just put the CustomResource in your CloudFormation stack:

 ---
 AWSTemplateFormatVersion: '2010-09-09'

 Resources:

   myBucketResource:
     Type: AWS::S3::Bucket
     Properties:
       BucketName: my-test-bucket-cleaning-on-delete

   cleanupBucketOnDelete:
     Type: Custom::cleanupbucket
     Properties:
       ServiceToken: arn:aws:lambda:eu-west-1:123456789012:function:clean-bucket-lambda
       BucketName: !Ref myBucketResource

Remember to attach a role to your lambda that has permission to remove objects from your bucket.

Furthermore keep in mind that you can create a lambda function that accepts CLI command line using the lambda function cli2cloudformation. You can download and install from here. Using that you just need to create a CustomResource like bellow:

"removeBucket": {
        "Type": "Custom::cli2cloudformation",
        "Properties": {
          "ServiceToken": "arn:aws:lambda:eu-west-1:123456789000:function:custom-lambda-name",
          "CliCommandDelete": "aws s3 rb s3://bucket-name --force",
        }
}
5
  • 6
    This is a great way to handle deleting the bucket from CloudFormation but I think the answer is just no - so I cannot mark this as the answer (but I did upvote it) - thanks
    – Jamie Czuy
    Commented Jul 11, 2017 at 15:40
  • 1
    More in-depth blog post on the topic: community.alfresco.com/community/platform/blog/2016/10/13/…
    – vincent
    Commented Dec 11, 2017 at 1:06
  • 1
    @JamieCzuy : This is a valid answer. You should mark it as resolver . Indeed, it is still in the context of cloudformation (CFN Custom resources) Commented Dec 19, 2017 at 18:01
  • There's a third party package that utilizes CDK and is available through NPM that does this, to save you writing your own: github.com/mobileposse/auto-delete-bucket
    – Jahorse
    Commented Feb 6, 2020 at 17:57
  • You can delete objects on bucket deletion: "To override this and force all objects to get deleted during bucket deletion, enable the autoDeleteObjects option." docs.aws.amazon.com/cdk/api/v2/docs/…
    – thoroc
    Commented Jul 12, 2022 at 7:47
8

I think your DependsOn is in wrong resource, at least it did not work for me properly because on stack deletion (via console), it would try to force bucket deletion first which will fail and then will attempt to delete custom resource, which triggers the lambda to empty the bucket. This will empty the bucket but the stack deletion will fail because it attempted to delete the bucket before it was empty. We want to initiate custom resource deletion first and then attempt to delete bucket after custom resource is deleted, so I did it like this and it works for me:

myBucketResource:
  Type: AWS::S3::Bucket
  Properties:
    BucketName: my-test-bucket-cleaning-on-delete

cleanupBucketOnDelete:
  Type: Custom::cleanupbucket
  Properties:
    ServiceToken: arn:aws:lambda:eu-west-1:123456789012:function:clean-bucket-lambda
    BucketName: my-test-bucket-cleaning-on-delete
  DependsOn: myBucketResource

This way you ensure the bucket deletion does not come first because there is another resource that depends on it, hence the depending resource is deleted first (which triggeres the lambda to empty the bucket) and then bucket is deleted. Hope someone finds it helpful.

2
  • 2
    I agree with this. You can also avoid the need for DependsOn, by making BucketName a reference to the Bucket, like this: BucketName: !Ref myBucketResource Commented Apr 25, 2019 at 10:09
  • That's true. I fixed it. =) Thanks Commented Jun 24, 2019 at 17:47
5

You should empty the bucket:

$ aws s3 rm s3://bucket-name --recursive

Then delete the Bucket

$ aws cloudformation delete-stack --stack-name mys3stack
1
  • 15
    The OP was asking if there is a way to NOT have to do that.
    – malana
    Commented Dec 10, 2020 at 11:30
5

botocore.vendored is deprecated and will be removed from Lambda after 2021/01/30.

Here is an update version

Type: AWS::Lambda::Function
Properties:
  Code: 
    ZipFile: 
      !Sub |
        import json, boto3, logging
        import cfnresponse
        logger = logging.getLogger()
        logger.setLevel(logging.INFO)

        def lambda_handler(event, context):
            logger.info("event: {}".format(event))
            try:
                bucket = event['ResourceProperties']['BucketName']
                logger.info("bucket: {}, event['RequestType']: {}".format(bucket,event['RequestType']))
                if event['RequestType'] == 'Delete':
                    s3 = boto3.resource('s3')
                    bucket = s3.Bucket(bucket)
                    for obj in bucket.objects.filter():
                        logger.info("delete obj: {}".format(obj))
                        s3.Object(bucket.name, obj.key).delete()

                sendResponseCfn(event, context, cfnresponse.SUCCESS)
            except Exception as e:
                logger.info("Exception: {}".format(e))
                sendResponseCfn(event, context, cfnresponse.FAILED)

        def sendResponseCfn(event, context, responseStatus):
            responseData = {}
            responseData['Data'] = {}
            cfnresponse.send(event, context, responseStatus, responseData, "CustomResourcePhysicalID")            

  Handler: "index.lambda_handler"
  Runtime: python3.7
  MemorySize: 128
  Timeout: 60
  Role: !GetAtt TSIemptyBucketOnDeleteFunctionRole.Arn    
0
3

Some won't like my answer, but here it is.

If your reason for using CloudFormation is to create and destroy S3 buckets in a development environment, use Terraform instead.

Why?

Terraform has an attribute named force_destroy.

Example: main.tf

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

provider "aws" {
  profile = "aws_profile_name"
  region  = "us-west-2"
}

resource "aws_s3_bucket" "public" {
  bucket        = "django-public-bucket"
  force_destroy = true

  tags = {
    Environment = "dev"
  }
}
2

It kind took a little bit to get it to work on python 3.8, so I am sharing it with the community.

Python 3.8 lambda doesn't support any longer from botocore.vendored import requests.

You can use the code below to inform Cloudformation.

import urllib

request = urllib.request.Request(event['ResponseURL'],
                                 method="PUT",
                                 data=json.dumps(response_body).encode('utf-8'),
                                 headers={'Content-Type': "''"})

with urllib.request.urlopen(request) as response:
    print("Status code: " + response.reason)

Another small note: when the lambda receive a create request from cloudformation you can put whatever unique value in the PhysicalResourceId response. When it is an UPDATE/DELETE you also receive that parameter from CloudFormation and you must reuse the value from the input in the response.

1

The least expensive solution I found is to utilize terraform to create CloudFormation template, and make use of destroy-time provisioner to empty S3 bucket before destroying CloudFormation stack. Example:

 resource "aws_cloudformation_stack" "this" {
  count        = 1
  name         = "stack_name"
  capabilities = ["CAPABILITY_IAM", ...]

  template_url = "The_URL_of_cloudformation_template"

  provisioner "local-exec" {
    command    = "${path.module}/resources/clean.sh"
    on_failure = continue
    when       = destroy

    environment = {
      BUCKET   = self.outputs["BucketName"]
    }
  }
}

And the script clean.sh should look like this script:

#!/bin/bash
aws s3 rm s3://${BUCKET} --recursive

And when you want to delete a CloudFormation stack, you must set the argument count to 0, because destroy-time provisioners can only run if they remain in the configuration at the time a resource is destroyed.

0

Updated answer: No, but you can use triggers to make lambdas do this for you before or after deployment changes: https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.triggers.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.