1

The AWS doc states

To allow your function time to process each batch of records, set the source queue's visibility timeout to at least six times the timeout that you configure on your function. The extra time allows for Lambda to retry if your function is throttled while processing a previous batch.

From my understanding, the Lambda polling process will invoke the Lambda function synchronously without any retry, why 6 ? Why not 2 ?

2
  • 3
    Presumably Lambda retries more than twice. So, let's say the Lambda timeout is 1 minute. If Lambda retried 3 times, then it would need the SQS timeout to be above 3 minutes. The recommendation of 6 suggests that Lambda might retry 5 times, or perhaps less but with a pause between the retries. Commented Nov 27, 2023 at 22:18
  • Thanks. I struggled to find documentation on that unfortunately :'(
    – John Doe
    Commented Nov 29, 2023 at 12:34

1 Answer 1

2

From my understanding, the Lambda polling process will invoke the Lambda function synchronously without any retry, why 6 ? Why not 2 ?

Here's what happens when your Lambda is called through the SQS subscription:

  1. The subscription polls the queue and retrieves a message (or a bunch of messages)
  2. SQS marks these messages as in-flight and starts the timer.
  3. The subscription tries to fire your lambda handler.
    1. In the past, the only way to throttle the subscription was to set the concurrency on the lambda level. The subscription would always poll for the messages, but it didn't know if it could process them after polling. In this case, if the subscription could not launch another instance of the handler because of the concurrency limits, the message would time out and fail.

    2. These days, you can set maximum concurrency on the subscription level, so that the subscription does not poll for the messages if it sees that it can not process them.

      Still, if the lambda is being called by something else, the concurrency issues will still happen.

  4. The handler gets the messages in its event argument and does its job. It does not communicate with SQS while doing that.
  5. If the handler runs longer than the message's lifetime, SQS returns the message back to the queue (the retry counter permitting). In this case, the message is getting polled again by the subscription and gets passed to another instance of the handler. The first handler doesn't know about that, and keeps processing the message.

As you can see, what we've got here is failure to communicate. Lambda, SQS and the subscription don't speak to each other while the message is being processed, and in the past didn't speak even before that.

There are two ways to deal with it:

  1. Put longer timeouts everywhere and hope that with enough retries, the message will eventually get processed. The number 6 was probably chosen because the retry count of 5 is recommended below, and the throttled handler processing the previous message could only time out five times, plus the time it takes to process the new message.

  2. Keep a thread in the lambda processor, extending the message visibility timeout in fixed small increments (say, 5 seconds) while the handler is running. It is not done by default, and requires custom code. Every timeout extension also counts as a request for the billing purposes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.