17

Currently, I have an AWS SQS as a trigger to my AWS Lambda function.

I would like to implement long polling to reduce costs since I've used up 70% of my monthly free tier, mostly from empty receives.

I tried setting up long polling by changing the queue attribute ReceiveMessageWaitTimeSeconds to 20 seconds:

SQS Details showing ReceiveMessageWaitTimeSeconds set to 20s

However, this didn't seem to reduce the number of empty receives, where the settings were changed on 11/19, between 2:00 - 3:00. Empty Receives graph shows the same trend even after setting long polling

According to the AWS Documentation, WaitTimeSeconds has priority over the queue attribute ReceiveMessageWaitTimeSeconds

Short polling occurs when the WaitTimeSeconds parameter of a ReceiveMessage request is set to 0 in one of two ways:

  • The ReceiveMessage call sets WaitTimeSeconds to 0.
  • The ReceiveMessage call doesn’t set WaitTimeSeconds, but the queue attribute ReceiveMessageWaitTimeSeconds is set to 0.

Note

For the WaitTimeSeconds parameter of the ReceiveMessage action, a value set between 1 and 20 has priority over any value set for the queue attribute ReceiveMessageWaitTimeSeconds.

Since AWS Lambda is receiving the SQS requests, I don't think WaitTimeSeconds can be configured.

Why doesn't my long polling configuration work in this situation? Am I misunderstanding something, or did I configure it wrong?

Thank you!

2
  • 1
    Why are you using Amazon SQS to trigger a Lambda function? You could instead send a message to Amazon SNS, which can also trigger a Lambda function but does not require polling. Is there a particular need for your application to use SQS? Nov 19, 2018 at 11:37
  • We prefer to use SQS for the letter retention period (max 14 days) and the easy setup for dead letter queues
    – naribo
    Nov 19, 2018 at 14:25

1 Answer 1

18

Actually Long Polling is working in your situation.

5 lambdas * polling / 20 seconds * 3600 seconds in an hour = 900 receives/hour

What I think you've missed is the "5 minimum concurrent lambdas". This is implied in the Lambda Scaling Behaviour documentation, but is more helpfully and explicitly laid out in the "Additional Information" section of the announcement/deep-dive blog.

When an SQS event source mapping is initially created and enabled, or when messages first appear after a period with no traffic, then the Lambda service will begin polling the SQS queue using five parallel long-polling connections. The Lambda service monitors the number of inflight messages, and when it detects that this number is trending up, it will increase the polling frequency by 20 ReceiveMessage requests per minute and the function concurrency by 60 calls per minute.

5
  • Thanks @thomasmichaelwallace! This explains it. So to clarify, Lambda long-polls by default, which explains why changing my queue attribute ReceiveMessageWaitTimeSeconds didn't change the number of empty receives?
    – naribo
    Nov 20, 2018 at 1:47
  • 1
    Yes- Lambda long polls (every 20) until there are messages, and then it moves to increasingly shorter polling until the queue is empty. (It doesn't respect the queue's RecieveMessageWaitTimeSecond). Nov 20, 2018 at 7:29
  • 1
    This is great, but do we have a solution how to reduce this cost? Oct 15, 2019 at 8:42
  • 5
    Unfortunately not, 15 long (3 times a minute; five in parallel) is the minimum. If you reserve concurrency you'll still make the polls, you'll just get throttled on lambda (ultimately costing you more). If your data is moving slow enough that SQS' cost is a problem you may be able to safely use SNS instead which does not have this flat "running" cost. Oct 15, 2019 at 9:04
  • Just adding to @thomasmichaelwallace's answer - you can use SNS instead of SQS, but there are some limitations, details are well described here : aws.plainenglish.io/…
    – Nigrimmist
    May 3, 2022 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.