We have application where multiple subscribers are writing to publisher Kafka topic This data is then propagated to specific subscriber topic then subscriber consumes this data from specific topic assigned to it.
We want to use SQS for same purpose but issue is we will again need an SQS for each subscriber. Handling these multiple SQS will create an issue and if there is time when no data is published to subscriber the queue assigned to it will be idle.
Is there any way i can use single SQS from which all subscribers can consumed messages base don message identifier. Challenges needs to be cover in this design are:
- Each subscriber can get its message based on identifier
- Latency must not be there in case one publisher publish very few messages and other one is publishing it in millions.
- We can have one SQS for each publisher but single SQS for all subscribers of this publisher.
Can any one suggest any architecture using similar implementation.
Thanks