0

I am looking at the Pipe streams tutorial. I need something similar, but only local.

In other words, the code in the example is:

final StreamsBuilder builder = new StreamsBuilder();
builder.stream("streams-plaintext-input").to("streams-pipe-output");

final Topology topology = builder.build();
final KafkaStreams streams = new KafkaStreams(topology, props);

which would take messages from streams-plaintext-input topic and put them into streams-pipe-output topic.

Is there a way to do this without creating an output topic? In other words, have each client use streams and just get the output locally?

The use case is basically to get messages and perform some filtering, grouping and such and then to consume that stream as if it were the original stream.

However, there are going to be a lot of consumers with different transformation criteria and I'd not like to have a topic for each of them, especially since this is temporary data, more like a continuous cache.

If there are some tutorials or examples that I can go through, please let me know.

2
  • 1
    Could you describe what you mean by then to consume that stream as if it were the original stream? If you need to transform data you just need to: builder.stream("streams-plaintext-input").filter(...).map(...) etc. Mar 7, 2019 at 21:50
  • Not sure what you mean by "locally" -- maybe KStream#foreach() is what you are looking for? -- Also, if you want to apply different processing, you can "reuse" the same KStream stream variable and "broadcast" the data into different pipeline: stream.filter(); ... stream.map() etc. Overall, you should read the docs and check out the examples (docs.confluent.io/current/streams/index.html and github.com/confluentinc/kafka-streams-examples) Mar 9, 2019 at 6:41

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.