7

I'm currently using DynamoDB streams to process changed collection values with lambda functions, however, currently, I'm only running two lambda instances in parallel, which is not enough to process all the incoming data and lambda functions are just queued up.

From aws documentation I can see that number of lambdas that can run in parallel is proportional to the number of shards of your DynamoDB:

If you create a Lambda function that processes events from stream-based services (Amazon Kinesis Streams or DynamoDB streams), the number of shards per stream is the unit of concurrency. If your stream has 100 active shards, there will be 100 Lambda functions running concurrently. Then, each Lambda function processes events on a shard in the order that they arrive.

So my question is, how do I increase the number of shards of my DynamoDB? Is it even possible? I couldn't find how to set it up in the settings.

1
  • I believe DynamoDB automatically shards based on the hash key assuming you have a hash and range key setup.
    – idbehold
    Feb 17, 2017 at 20:18

2 Answers 2

1

No, its not possible to manually control number of shards in DDB UpdateStream. DDB automatically handles that for you by creating as many shards to match the incoming rate of updates.

Ideally updates happening to your DDB table is supposed to flow through some shard (updates happening to same record will always go to same shard meaning they are partitioned based on your hashKey). It is your stream of updates that too in chronological order thus updates over same record end up (or say queued up) in same shard so that end processor process them in sequence they happened.

Each shard has its own throughput capacity for in and out of data unless there is need of more shards to support in coming rate of updates on table (which in case of DDB updates streams is high write tps on your table, which current number of shards can't handle)

4
  • I may be doing something wrong because it seems incredibly slow to me. I am able to write data into the DB fairly fast but it can take much much longer to receieve all of the events from the stream, because it appears to only invoke a single lambda at a time. My primary partition key is uuid() for each item, so I don't see why it wouldn't have good shard distribution. You said "DDB automatically handles that for you by creating as many shards to match the incoming rate of updates." But this rate is too slow and I would like it to automatically create more shards :( Nov 30, 2017 at 23:15
  • For simplicity I mentioned hashKey is direct shard mapping, In reality its little more than that its your compositeKey and parallel updates on table across compositeKey distribution to balance load across shards. So if you are updating few compositeKeys only and they happen to fall under same shard then they will run in sequential order not parallel. Point is multiple CompositeKeys can fall under same shard and its all dynamic based on your keys and table load distribution. DDB streams is not abt processing table updates in parallel, its abt processing table updates in their occurrence order Dec 14, 2017 at 23:05
  • Clarification for "DDB streams is not abt processing table updates in parallel, its abt processing table updates in their occurrence order" Updates getting in same shard gets processed sequentially (updates in same shard are in chronological order) while multiple shards can trigger lambdas in parallel as updates in different shards are independent of each other. Dec 15, 2017 at 19:05
  • It seems a mistake was made early on regarding the keys not being random enough and so I'm in a position where it seems like sharding isn't happening enough. So I'm going to have to shift the value being used as a key into another field and then alter the keys to be more random. Its a fairly painful thing to have to fix later for performance reasons. In my case I don't really need sequence, though the keys aren't randomly distributed very well (think product skus). Dec 16, 2017 at 15:36
0

Enabling autoscaling in the DynamoDB tables helps with auto-sharding. If you have specified provisioned capacity for the tables, the sharding becomes slightly rigid and might not be suitable for scaling. It may be more suitable if your events are sequential and more control is required.

To enable auto-scaling for your tables, go to the console, click on your DynamoDB table and click on the Capacity tab. There you can choose the ranges based on your forecasted traffic.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.