How can I reduce the latency that I'm currently seeing between Lambda & DynamoDB?
It's a Java lambda and it's taking 4s to perform each of 2 DynamoDB operations, using the AWS-supplied SDK. I've heard that these typically complete within <20ms, but for me it's 3 orders of magnitude higher. These long operations are (a) creating the DynamoDB object and (b) performing a table scan on a completely empty table (code below).
What should I do to reduce the latency?
Things I've tried
- Both the lambda and DynamoDB are in the same region (eu-west-1).
- The table has 5 RCU & WCU. Increasing these doesn't help.
- The maximum memory used by the lambda is 92MB. If I allocate the minimum 128MB then it times out after 15s. Increasing the memory to 512MB achieves the timings of 4s per call and increasing it again to 1GB reduces it to 2s per call. However, this is an absurd amount of memory for a trivial lambda and still leaves me with >200x the expected latency.
- Table metrics show the table scan time is between 12ms and 15ms. This is what I was expecting. Even with the overhead of establishing a network connection, I'm still expecting a few 10s of milliseconds, not several seconds.
- I'm triggering the lambda using the test function in the AWS console. I've also tried triggering via API Gateway (which is what I'll be doing eventually) with the same results.
- I've tried calling the lambda several times in quick succession (to reduce the likelihood that I'm suffering setup costs). It didn't help.
- Logging shows all other parts of the lambda running very quickly (milliseconds).
Code snippets
Creating the DynamoDB objects
log("Creating AmazonDynamoDB");
AmazonDynamoDB db = AmazonDynamoDBClientBuilder
.standard()
.withRegion(Regions.EU_WEST_1)
.build();
log("Creating DynamoDBMapper");
DynamoDBMapper mapper = new DynamoDBMapper(db);
Performing the scan
log("Scanning table");
List<MyItem> items = dbMapper.scan(MyTable.class, new DynamoDBScanExpression());
for (MyItem item : items) {
// Irrelevant - there aren't any
}
log("Table scan complete");
Sample logs
And here are the logs from a run.
20:07:41 START RequestId: 9d436db7-5d32-11e8-8555-8564d2094ccc Version: $LATEST
20:07:41 Received request: APIGatewayRequest(path=/data/foo, httpMethod=POST, body=)
20:07:41 Creating AmazonDynamoDB
20:07:45 Creating DynamoDBMapper
20:07:45 Creating DataHandler
20:07:45 Handling request
20:07:45 Scanning table
20:07:49 Table scan complete
20:07:49 Request handled - response object: []
20:07:49 APIGatewayResponse(isBase64Encoded=false, statusCode=200, body=[], headers={})
20:07:49 END RequestId: 9d436db7-5d32-11e8-8555-8564d2094ccc
20:07:49 REPORT RequestId: 9d436db7-5d32-11e8-8555-8564d2094ccc Duration: 8256.47 ms Billed Duration: 8300 ms Memory Size: 512 MB Max Memory Used: 85 MB