0

I have a client that is constantly pouring semi-real-time data into an sqs queue, and want to process and store the messages. My first thought was to use a CloudWatch scheduler that prompts a Lambda with the approximate number of messages that lambda then spawns worker lambdas to process and push the data into a Firehose. The problem is that there will be hundreds of thousands of messages put into the queue every day. I could also use EC2 to do this, but is there any other cost-effective way to process the queue semi-real-time.

6
  • 2
    How much processing does each message require? A t2.nano costs USD $0.14/day. Lambda will end up being much more expensive than a small EC2.
    – stdunbar
    May 14, 2018 at 16:40
  • @stdunbar It takes around .012 seconds to process one message. The EC2 seems to be the way to go. Thank you.
    – j doe
    May 14, 2018 at 16:43
  • 3
    May be SQS is not the right service to do this. Since it seems your data is like a stream, you may want to look at Kinesis.
    – Asdfg
    May 14, 2018 at 16:45
  • @Asdfg I would have loved to use Kinesis, but the client only dumps the data into SQS for some reason. SQS is not the best resource for this task.
    – j doe
    May 14, 2018 at 16:47
  • An ecs service with app autoscaling based on the queue length could work pretty well for sqs if a stream is not an option.
    – khaleesi
    May 14, 2018 at 17:05

1 Answer 1

1

The recommended solution for processing streaming data in AWS Lambda is to send the data to Amazon Kinesis, which can then trigger a Lambda function automatically. Kinesis also preserves the ordering of messages. (Amazon SQS only preserves ordering if you use a FIFO queue, which has throughput limitations.)

If you really are limited to processing from SQS, you could write a program that pulls from SQS and pushes to Kinesis or simply pull from SQS and process the data immediately. Such a program could run on an Amazon EC2 instance, or could be triggered on a regular basis by a scheduled Amazon CloudWatch Event.

The main thing to consider is how to handle variable volumes. If you cannot accept long delays between messages arriving and being processed, you will need to either use Lambda (automatically scalable) or have plenty of available processing power to handle the spikes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.