0

I am using Kafka to decouple my services, but I'm having some seconds thoughts on the way services consume and produce inputs and outputs.

If I have a service A, which takes data from some external service out of my control, I am forced to adapt to the data format (domain) that external system provides. Following such practice, my service A pushes its results to a topic in its own format (domain).

Incidentally, I have a service B, which does similar thing to service A, but consumes some other external service, and has its own data format (domain), which it pushes to a separate topic.

Now, the semantics of the data produced by A and B are similar, but not the same. But, next step in the pipeline is a service C, which should consume both what A and B produce, do something with it and spit out the results.

Should C only know how to consume data from one place, which would imply that A & B (and any other ones in the future) need to produce their outputs in C-specific domain? That would mean, if the C consumer ever changes it's domain, A, B, and any other producers will have to change, which I don't like. Also, if I add another consumer, D, for example, that means that A and B, using this analogy, should know that D is also their consumer, which looks horrible to me.

I was thinking that C should be responsible for it's inputs, meaning it has a dependency on A and B models (and any other that might produce their own data). It also implies that, when a new source is added, C must be changed to include that data as well.

Effectively, I'm leaning towards ManySources-OneSink component, instead of OneSource-ManySinks.

Are there any preferred practices?

4 Answers 4

4

What you are talking about is context mapping. The relationships between BCs. The upstream and downstream sides. It's not your choice in relationship between two BCs to decide which of the two is the upstream and the downstream. They are per se. You can choose how to integrate (use REST API, use events, sync, async, ...). You should draw a context map, identify which strategic pattern applies to each relationship between BCs, and decide how to implement it.

2

Before you choose an integration mode, there's analysis to be done on a strategic level about the business subdomains, teams that will maintain these services, who owns what, how you expect the models to evolve, etc.

I'd recommend having a look at the Context Mapping part of the DDD book and then Enterprise Integration Patterns by Hohpe and Woolf for concrete implementation.

1

Are there any preferred practices?

Message specifications.

That is, instead of coupling A, B, and C to each other, you could them to message formats mA, mB, and mC. So long as (mA, mB, mC) are compatible, the services will be able to communicate with each other.

One way to achieve this is to restrict mA, mB, and mC to being different "versions" of the same schema, where the evolution of the schema is constrainted

  • No new required fields may be added after the initial version
  • Optional fields should have default values
  • A consumer should ignore an unrecognized field
  • Fields can be deprecated, but they are never re-used (for example, see HTTP status code 306)

Greg Young's book Versioning in an Event Sourced System dedicates a chapter to this idea. You'll see similar ideas is you look at the various standardized message serializations (Avro, Protocol Buffers, etc).

Should one service produce output on its own topic and make dependent services to consume from them, or vice versa, or some third option?

Mostly, that's plumbing -- how do we get a copy of a message from one system to another?

Conceptually, it seems that you want C to be consuming a logical outer join of the messages produced by A and B. So I suppose the immediate question would be whether you currently have consumers of these same messages from A that don't want the messages from B, or vice versa. If this is the only use case you know about, there might be benefits to reducing things to a single topic.

1
  • And what about the sources/sinks? Should one service produce output on its own topic and make dependent services to consume from them, or vice versa, or some third option?
    – ioreskovic
    Oct 8, 2018 at 14:27
0

Each of your services should use the data format that makes the most sense for its specific business logic.

For integration purposes:

  • Inbound channel adapters - converts data to the desired format as it enters the service.
  • Outbound channel adapters - converts data to the desires format as it exits the service.

For more information: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ChannelAdapter.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.