1

I'm pretty new to the Akka streams technology and I didn't find any source of information regarding my use case.

I want to use Akka streams to consume messages from a Kafka topic, process them (while using another data source), and store the output in a mongo collection.

We are working in a methodology of a topic (with one partition) and collection per organization in order to achieve parallelism. The topics and collections can be created dynamically and the application needs to be highly scalable and deployed on multiple pods.

What are the best practices in order to achieve this behavior? Is it possible to do with Akka streams? Thanks!

1
  • One option is to use Kafka Connector that MongoDB has written for your output. Mar 16, 2022 at 16:51

1 Answer 1

0

Given that you're doing topic and collection per organization, the most applicable pattern is probably to use Akka Cluster to form a collaborating set of pods. A cluster singleton maintains the topic/collection to organization mapping; the data for that singleton (which organizations are you consuming for and what topic and collection corresponds to that organization) could be fed from a Kafka topic, or the cluster singleton could store that state.

The process of running the consumption streams would then be managed by actors which are sharded across the cluster by organization. I'd suggest using Alpakka Kafka and Alpakka's MongoDB integration for implementing the stream. Pay particular attention to the message delivery guarantees in Alpakka Kafka: that will drive your decisions around storing Kafka offsets (basically every conceivable guarantee is possible).

The cluster singleton's responsibility is then to periodically ping the sharded actors (aka "entities") to make sure that they're running.

Akka Cluster (in conjunction with things like the split-brain resolver) will handle things like adding pods, instance/network failures etc.

1
  • Thanks for the detailed answer! I'll learn more about Akka clusters
    – oy121
    Mar 20, 2022 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.