5

I'm quite new to Kinesis Streams and have spent a bunch of time wading through documentation but haven't been able to determine if it's possible to have multiple "topics" (for lack of a better term) on a single shard. Originally I thought that this is what partition keys were for, however after reading this:

A partition key is used to group data by shard within a stream. The Streams service segregates the data records belonging to a stream into multiple shards, using the partition key associated with each data record to determine which shard a given data record belongs to.

...it sounds to me like the partitions (can) span multiple shards.

I currently don't require more than a single shard but I do need to be able to separate messages into different channels/topics within that shard- is this not possible with Kinesis?

-- EDIT --

Worth mentioning- am using both the KCL and the KPL in my application.

1 Answer 1

4

With only a single shard you won't be able to separate your data into separate channels. To consume the records you'll have to call GetShardIterator first. GetShardIterator takes the stream name, shard id and an iterator type.

The returned shard iterator is your pointer into the stream. Kinesis will return records to you if you call GetRecords, passing in the shard iterator. The response will contain a batch of records including the data, a timestamp, a sequence number and the partition key.

Today there is no way to specify a topic or any other filter when calling GetRecords/GetShardIterator.

1
  • 3
    Hey Benno, it is already 2020, do we have any updates on this? :) Still have the same question and wondering if something changed since 2016. Tried to find some info but to no success. Jan 31, 2020 at 22:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.