56

I know how to expire objects in an S3 bucket using object expiration rules given a certain prefix, however for my purposes, I would like to set the expiry date programmatically on a per object basis.

The Java SDK seems to indicate that this is possible as it has a setExpirationTime method, however whenever I set an expiration Date using this method, nothing seems to happen and the object never expires. Additionally, looking at the object properties through the aws console, no expiry appears to be set.

Is per file expiration not supported ? / Are there any extra steps I need to do to get it to work ? / If per file expiration is not supported, is it possible to exclude a file that matches an expiration prefix from being expired ?

Thanks in advance!

7 Answers 7

26

It doesn't look like per-object expiration is supported, but rather a per-bucket lifecycle configuration with up to 100 rules per configuration, as you have found.

A bucket has one lifecycle configuration. A lifecycle configuration can have up to 100 rules.

The lifetime value must be a nonzero positive integer. Amazon S3 calculates expiration time by adding the expiration period specified in the rule to the object creation time and rounding the resulting time to the next day midnight UTC.

If per file expiration is not supported, is it possible to exclude a file that matches an expiration prefix from being expired ?

It doesn't look like you can overlap rules, either.

Take care to ensure the rules don't overlap. For example, the following lifecycle configuration has a rule that sets objects with the prefix "documents" to expire after 30 days. The configuration also has another rule that sets objects with the prefix "documents/2011" to expire after 365 days. In this case, Amazon S3 returns an error message.

4
  • 1
    This makes sense, but do you know what the setExpirationTime method is for then?
    – Mac Adada
    Aug 29, 2012 at 21:38
  • I believe it is just a property setter for the ObjectMetadata wrapper in the java SDK. It is called with the results of the inernal putObject invoke call. So it just sets the property so you can look at it in higher level objects. Setting it yourself will do nothing. Aug 29, 2012 at 22:02
  • 2
    What about the --expires parameter on the AWS CLI? May 7, 2015 at 9:52
  • 3
    I believe the per-object "expiration time" property you are talking about refers to cache expiration, as with the HTTP "Expires" header: tools.ietf.org/html/rfc2616#section-14.21 It will not delete the S3 object.
    – aib
    Jun 15, 2017 at 11:44
18

An alternative way to accomplish this task is to make use of object tags.

Set a unique tag for each object of the objects you want to set lifetimes for, Then create a life cycle configuration rule for each object and in the filter element you can use the object specific tag.

This tag can be the object name, Also you have the ability to make a rule for a subset of objects that don't have common prefix.

You can find more info about life cycle configuration here http://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html

1
  • can we update the prefix based expiration rule that has been applied to an object to tag based expiration ?? I couldn't able to update it. But for the objects that are not matched with prefix/ based filter, I could able to apply and update expiration by applying different tags. May 12, 2020 at 12:04
12

Here's an alternative approach that might work for some situations:

You could set the bucket lifecycle to expire objects after a short number of days (1, 7, whatever). Then, before each object expires, programmatically extend its expiration if you don't want that object to expire yet.

You can reset the timestamp on an S3 object by "copying" the object to itself with an instruction to change the storage class to the same class it currently is. It's basically a "noop" that acts like a Linux/Unix "touch" command.

I wrote an article that walks through example aws-cli commands demonstrating how to do this: http://alestic.com/2013/09/s3-lifecycle-extend

2
  • 1
    Very innovative solution, but, "there's gotta be a better way". Or, I guess in this case, AWS should provide a better way.
    – AJB
    Oct 16, 2016 at 7:49
  • 1
    Indeed creative work around but you gotta be sure your program that "touch" each objects for which you want to extend their life execute because if for whatever reason that program doesn't run, you could lose data. Mar 29, 2018 at 0:16
6

Similarly to Eric's answer you could set a short bucket lifecycle policy and then use retention period locks (see the docs) to set expiration dates on a per-object basis.

4

To expire objects inside S3 bucket you need to use LifeCycle rules.

To expire some objects, you must use filters of the life cycle rule. I attach screenshot of AWS console to see the available options.

You have 3 ways to filter out the objects:

  1. By prefix of the path to the object inside bucket
  2. By tags added on the object (you should add tags while creating the objects to make it useful. You can add tags later too though.)
  3. By size of the objects

While you can not specify exact date for each object, you can specify tag filter and how old the object should be to apply the life cycle rule. So what I do is I create defined set of tags e.g.: expire-7d, expire-1d etc. And then create set of life cycle rules for each tag. So 1st rule would take any object which is 7 days old and has such tag and would remove them.

To remove objects, you would need to select rule action:

Expire current versions of objects

enter image description here

Yes, you can combine multiple filters on single life cycle rule

enter image description here

If want to use API - here are docs:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html

The most important part is LifecycleRuleFilter

https://docs.aws.amazon.com/AmazonS3/latest/API/API_LifecycleRuleFilter.html

1

I extended the expiration time by copying the object into itself using s3cmd command. It basically worked like 'touch'.

s3cmd cp s3://bucket-name/file.txt s3://bucket-name/file.txt

0

You could split the files into 'folders' based on their rules and apply Prefix Filter.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.