7

We seem to be deterministically encountering this problem and aren't sure where we're misconfigured. For lambdas running less than ~5 minutes, our invocation succesfully wraps up ~0.5 seconds after the lambda completes. However for anything running longer than that, we can see that the lambda completes in the lambda logs, but our client invocation throws a ClientExecutionTimeoutException after 15 minutes.

After encountering the problem with other (otherwise successful) lambdas, we created a basic test lambda on Node with a sleep function and have been able to deterministically reproduce the issue:

function sleep(s) {
  return new Promise(resolve => setTimeout(resolve, s * 1000));
}
const sleepMinutes = 60 * 5;
exports.handler = async (event) => {
    console.log(`received lambda invocation, sleeping ${sleepMinutes}`);
    const response = {
        statusCode: 200,
        body: JSON.stringify(`finished running, slept for ${sleepMinutes} minutes`),
    };
    await sleep(sleepMinutes);
    console.log('finished sleeping');
    return response;
};

Our lambda invocation client is using these client configs:

clientConfig.setRetryPolicy(PredefinedRetryPolicies.NO_RETRY_POLICY);
clientConfig.setMaxErrorRetry(0);
clientConfig.setSocketTimeout(15 * 60 * 1000);
clientConfig.setRequestTimeout(15 * 60 * 1000);
clientConfig.setClientExecutionTimeout(15 * 60 * 1000);

Is there a ~5 minute timeout config we're missing?

4
  • are you able to see 'finished sleeping' in logs ? Dec 12, 2019 at 18:01
  • yes. looking at the lambda logs, everything seems to wrap up appropriately
    – dev_feed
    Dec 12, 2019 at 18:42
  • in further testing, I was able to reproduce the issue by invoking lambdas from the CLI. on a lambda that runs for 8 minutes and otherwise succeeds, the command failed in about ~6 minutes. seems like maybe there is a network config ending this tcp connection?
    – dev_feed
    Jan 8, 2020 at 18:29
  • this is from /etc/os-release: NAME="Amazon Linux AMI" VERSION="2017.03" ID="amzn" ID_LIKE="rhel fedora" VERSION_ID="2017.03" PRETTY_NAME="Amazon Linux AMI 2017.03" ANSI_COLOR="0;33" CPE_NAME="cpe:/o:amazon:linux:2017.03:ga" HOME_URL="aws.amazon.com/amazon-linux-ami"
    – dev_feed
    Jan 8, 2020 at 18:29

3 Answers 3

7
+100

Javadocs in aws-sdk-java says:

 For functions with a long timeout, your client might be disconnected during synchronous invocation while it waits for a response. Configure your HTTP client, SDK, firewall, proxy, or operating system to allow for long connections with timeout or keep-alive settings.

On the other hand, previously AWS Lambda was limited up to 5 minutes, later this limit was increased up to 15 minutes.

I would check:

  1. The client sdk version is up to date
  2. The connection is not closed by your network
  3. Move to an async invocation via AWSLambdaAsyncClient.invokeAsync() for long running invocations.
2
  • 1
    to clarify, the issue is not that my client is returning before the lambda is finished. the issue is that for a lambda running longer than ~5 minutes, the lambda completes but my client will keep waiting for a response until the 15 minute timeout occurs
    – dev_feed
    Jan 8, 2020 at 15:48
  • we're on java sdk version 1.11.616, and the asynchronous invocation doesn't fit our use case for other reasons
    – dev_feed
    Jan 8, 2020 at 15:49
2

I've accepted Ezequiel's answer since it was technically a networking / OS issue, but here is a more detailed result:

We had to ensure all relevant clients were configured to keep alive tcp connections. We then had to add these properties to the /etc/sysctl.conf file on our EC2 residing in a private subnet because the NAT gateway is set to kill idle connections beyond 350s:

net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 100
net.ipv4.tcp_keepalive_probes = 6
1

I have experienced such errors. The problem is with the lambda context. Your function might not return the success but return fail, as you didn't finish the function with context success. Please check if you have done this when you finish the lamba.

Thanks.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.