From my understanding, the Lambda polling process will invoke the Lambda function synchronously without any retry, why 6 ? Why not 2 ?
Here's what happens when your Lambda is called through the SQS subscription:
- The subscription polls the queue and retrieves a message (or a bunch of messages)
- SQS marks these messages as in-flight and starts the timer.
- The subscription tries to fire your lambda handler.
In the past, the only way to throttle the subscription was to set the concurrency on the lambda level. The subscription would always poll for the messages, but it didn't know if it could process them after polling. In this case, if the subscription could not launch another instance of the handler because of the concurrency limits, the message would time out and fail.
These days, you can set maximum concurrency on the subscription level, so that the subscription does not poll for the messages if it sees that it can not process them.
Still, if the lambda is being called by something else, the concurrency issues will still happen.
- The handler gets the messages in its event argument and does its job. It does not communicate with SQS while doing that.
- If the handler runs longer than the message's lifetime, SQS returns the message back to the queue (the retry counter permitting). In this case, the message is getting polled again by the subscription and gets passed to another instance of the handler. The first handler doesn't know about that, and keeps processing the message.
As you can see, what we've got here is failure to communicate. Lambda, SQS and the subscription don't speak to each other while the message is being processed, and in the past didn't speak even before that.
There are two ways to deal with it:
Put longer timeouts everywhere and hope that with enough retries, the message will eventually get processed. The number 6 was probably chosen because the retry count of 5 is recommended below, and the throttled handler processing the previous message could only time out five times, plus the time it takes to process the new message.
Keep a thread in the lambda processor, extending the message visibility timeout in fixed small increments (say, 5 seconds) while the handler is running. It is not done by default, and requires custom code. Every timeout extension also counts as a request for the billing purposes.