8

For the last few years we have used our own RM Application to process events related to our applications. This works by polling a database table every few minutes, looking for any rows that have a due date before now, and have not been processed yet.

We are currently making the transition to SNS, with SQS Worker tiers processing them. The problem with this approach is that we can't future date our messages. Our applications sometimes have events that we don't want to process until a week later.

Are there any design approaches, alternative services, clever tricks we could employ that would allow us to do achieve this?

One solution would be to keep our existing application running, at a simplified level, so all it does is send the SNS notifications when they are due, but the aim of this project is to try and do away with our existing app.

3 Answers 3

4

The database approach would be the wisest, being careful that each row is only processed once.

Amazon Simple Notification Service (SNS) is designed to send notifications immediately. There is no functionality for a delayed send (although some notification types are retried if they fail).

Amazon Simple Queue Service (SQS) does have a delay feature, but only up to 15 minutes -- this is useful if you need to do some work before the message is processed, such as copying related data to Amazon S3.

Given that your requirement is to wait until some future arbitrary time (effectively like a scheduling system), you could either start a process and tell it to sleep for a certain amount of time (a bad idea in case systems are restarted), or continue your approach of polling from a database.

If all jobs are scheduled for a distant future (eg at least one hour away), you theoretically only need to poll the database once an hour to retrieve the earliest scheduled time.

2
  • My situation is the same as the first paragraph of the question. You said that is the wisest approach. Could you clarify the design philosophy behind this structure and maybe explain any other best practices? I too was looking at SNS but I didn't find the scheduling feature I wanted. Now I'm looking at doing a simple "alerting app" in addition to my main app that just goes through the DB and fires off alerts. Is this really the way enterprise apps do it?
    – trademark
    Feb 16, 2018 at 5:51
  • @trademark, Please ask a new question rather than asking via a comment on an old question. Feb 16, 2018 at 22:33
1

A week might be too long as SQS message retention itself is only 15 days. If you are okay with maximum retention of 15days, one idea is to keep the changing the visibility of a message every time you receive until it is ready for processing. The maximum allowed visibility timeout is 12 hours. More on visibility timeout and APIs for changing them,

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html

1
  • 1
    It works, however it does feel like a hacky solution to get around the AWS imposed limits of 15 mins on the delayseconds and 12 hours on the visibility timeout. Oct 19, 2021 at 10:34
1

I found this approach: https://github.com/alestic/aws-sns-delayed. Basically, you can use a step function with a wait step in there

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.