Is there any way to force CloudFormation to delete a non-empty S3 Bucket?
8 Answers
You can create a lambda function to clean up your bucket and invoke your lambda from your CloudFormation stack using a CustomResource.
Below a lambda example cleaning up your bucket:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import json
import boto3
from botocore.vendored import requests
def lambda_handler(event, context):
try:
bucket = event['ResourceProperties']['BucketName']
if event['RequestType'] == 'Delete':
s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket)
for obj in bucket.objects.filter():
s3.Object(bucket.name, obj.key).delete()
sendResponseCfn(event, context, "SUCCESS")
except Exception as e:
print(e)
sendResponseCfn(event, context, "FAILED")
def sendResponseCfn(event, context, responseStatus):
response_body = {'Status': responseStatus,
'Reason': 'Log stream name: ' + context.log_stream_name,
'PhysicalResourceId': context.log_stream_name,
'StackId': event['StackId'],
'RequestId': event['RequestId'],
'LogicalResourceId': event['LogicalResourceId'],
'Data': json.loads("{}")}
requests.put(event['ResponseURL'], data=json.dumps(response_body))
After you create the lambda above, just put the CustomResource in your CloudFormation stack:
---
AWSTemplateFormatVersion: '2010-09-09'
Resources:
myBucketResource:
Type: AWS::S3::Bucket
Properties:
BucketName: my-test-bucket-cleaning-on-delete
cleanupBucketOnDelete:
Type: Custom::cleanupbucket
Properties:
ServiceToken: arn:aws:lambda:eu-west-1:123456789012:function:clean-bucket-lambda
BucketName: !Ref myBucketResource
Remember to attach a role to your lambda that has permission to remove objects from your bucket.
Furthermore keep in mind that you can create a lambda function that accepts CLI command line using the lambda function cli2cloudformation. You can download and install from here. Using that you just need to create a CustomResource like bellow:
"removeBucket": {
"Type": "Custom::cli2cloudformation",
"Properties": {
"ServiceToken": "arn:aws:lambda:eu-west-1:123456789000:function:custom-lambda-name",
"CliCommandDelete": "aws s3 rb s3://bucket-name --force",
}
}
-
6This is a great way to handle deleting the bucket from CloudFormation but I think the answer is just no - so I cannot mark this as the answer (but I did upvote it) - thanks Commented Jul 11, 2017 at 15:40
-
1More in-depth blog post on the topic: community.alfresco.com/community/platform/blog/2016/10/13/…– vincentCommented Dec 11, 2017 at 1:06
-
1@JamieCzuy : This is a valid answer. You should mark it as resolver . Indeed, it is still in the context of cloudformation (CFN Custom resources) Commented Dec 19, 2017 at 18:01
-
There's a third party package that utilizes CDK and is available through NPM that does this, to save you writing your own: github.com/mobileposse/auto-delete-bucket– JahorseCommented Feb 6, 2020 at 17:57
-
You can delete objects on bucket deletion: "To override this and force all objects to get deleted during bucket deletion, enable the autoDeleteObjects option." docs.aws.amazon.com/cdk/api/v2/docs/…– thorocCommented Jul 12, 2022 at 7:47
I think your DependsOn is in wrong resource, at least it did not work for me properly because on stack deletion (via console), it would try to force bucket deletion first which will fail and then will attempt to delete custom resource, which triggers the lambda to empty the bucket. This will empty the bucket but the stack deletion will fail because it attempted to delete the bucket before it was empty. We want to initiate custom resource deletion first and then attempt to delete bucket after custom resource is deleted, so I did it like this and it works for me:
myBucketResource:
Type: AWS::S3::Bucket
Properties:
BucketName: my-test-bucket-cleaning-on-delete
cleanupBucketOnDelete:
Type: Custom::cleanupbucket
Properties:
ServiceToken: arn:aws:lambda:eu-west-1:123456789012:function:clean-bucket-lambda
BucketName: my-test-bucket-cleaning-on-delete
DependsOn: myBucketResource
This way you ensure the bucket deletion does not come first because there is another resource that depends on it, hence the depending resource is deleted first (which triggeres the lambda to empty the bucket) and then bucket is deleted. Hope someone finds it helpful.
-
2I agree with this. You can also avoid the need for DependsOn, by making BucketName a reference to the Bucket, like this: BucketName: !Ref myBucketResource Commented Apr 25, 2019 at 10:09
-
You should empty the bucket:
$ aws s3 rm s3://bucket-name --recursive
Then delete the Bucket
$ aws cloudformation delete-stack --stack-name mys3stack
-
15The OP was asking if there is a way to NOT have to do that.– malanaCommented Dec 10, 2020 at 11:30
botocore.vendored is deprecated and will be removed from Lambda after 2021/01/30.
Here is an update version
Type: AWS::Lambda::Function
Properties:
Code:
ZipFile:
!Sub |
import json, boto3, logging
import cfnresponse
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
logger.info("event: {}".format(event))
try:
bucket = event['ResourceProperties']['BucketName']
logger.info("bucket: {}, event['RequestType']: {}".format(bucket,event['RequestType']))
if event['RequestType'] == 'Delete':
s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket)
for obj in bucket.objects.filter():
logger.info("delete obj: {}".format(obj))
s3.Object(bucket.name, obj.key).delete()
sendResponseCfn(event, context, cfnresponse.SUCCESS)
except Exception as e:
logger.info("Exception: {}".format(e))
sendResponseCfn(event, context, cfnresponse.FAILED)
def sendResponseCfn(event, context, responseStatus):
responseData = {}
responseData['Data'] = {}
cfnresponse.send(event, context, responseStatus, responseData, "CustomResourcePhysicalID")
Handler: "index.lambda_handler"
Runtime: python3.7
MemorySize: 128
Timeout: 60
Role: !GetAtt TSIemptyBucketOnDeleteFunctionRole.Arn
Some won't like my answer, but here it is.
If your reason for using CloudFormation is to create and destroy S3 buckets in a development environment, use Terraform instead.
Why?
Terraform has an attribute named force_destroy
.
Example: main.tf
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
}
}
}
provider "aws" {
profile = "aws_profile_name"
region = "us-west-2"
}
resource "aws_s3_bucket" "public" {
bucket = "django-public-bucket"
force_destroy = true
tags = {
Environment = "dev"
}
}
It kind took a little bit to get it to work on python 3.8, so I am sharing it with the community.
Python 3.8 lambda doesn't support any longer from botocore.vendored import requests
.
You can use the code below to inform Cloudformation.
import urllib
request = urllib.request.Request(event['ResponseURL'],
method="PUT",
data=json.dumps(response_body).encode('utf-8'),
headers={'Content-Type': "''"})
with urllib.request.urlopen(request) as response:
print("Status code: " + response.reason)
Another small note: when the lambda receive a create
request from cloudformation you can put whatever unique value in the PhysicalResourceId response. When it is an UPDATE/DELETE you also receive that parameter from CloudFormation and you must reuse the value from the input in the response.
The least expensive solution I found is to utilize terraform to create CloudFormation template, and make use of destroy-time provisioner to empty S3 bucket before destroying CloudFormation stack. Example:
resource "aws_cloudformation_stack" "this" {
count = 1
name = "stack_name"
capabilities = ["CAPABILITY_IAM", ...]
template_url = "The_URL_of_cloudformation_template"
provisioner "local-exec" {
command = "${path.module}/resources/clean.sh"
on_failure = continue
when = destroy
environment = {
BUCKET = self.outputs["BucketName"]
}
}
}
And the script clean.sh
should look like this script:
#!/bin/bash
aws s3 rm s3://${BUCKET} --recursive
And when you want to delete a CloudFormation stack, you must set the argument count
to 0, because destroy-time provisioners can only run if they remain in the configuration at the time a resource is destroyed.
Updated answer: No, but you can use triggers to make lambdas do this for you before or after deployment changes: https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.triggers.html