1

Imagine you have an architecture with one producer (P1) and many consumers (C1-C2-C3). When a small java client produces messages as M1, M2, M3 in order and another java clients (3x scaled to another machines) gets a message then writes the message to database table after calculating something.

What if calculation periods are different in consumer applications and the message which is consumed at first may be written to same table in last order, it probably cause data inconsistency.

Maybe I missed something in docs, but I wonder that how can kafka handles consistency in that scenario.

3 Answers 3

2

The consumers don't listen to the producer. Instead:

  1. The producer writes a message to a Kafka topic managed by the Kafka server cluster,
  2. A Kafka server persists that message in one of the partitions created for that topic and
  3. Only then do the consumers get access to the message.

If the consumers are in the same consumer group, then only one of them will be reading from the message's partition and only that consumer will read be able to read that message. If the consumers are not in the same consumer group then they may all be able to read the message. In fact, that message may be read many times by many consumers until the Kafka server deletes the message for being older than the configured time-to-live for the topic.

Once a consumer has read a message from a Kafka topic, Kafka has no control over how, when or even if that message is processed.

0
1

If you want to keep the order for relevant messages that you send to a Kafka topic, you could select one unique identifier regarding those messages as Kafka partition key.

For example, if you are processing transactions from different customers, you could select customerId(assuming it is an unique identifier for a customer) as partition key, so that all the messages that you sent to Kafka for a given customer will end up in the same partition; meaning that they will be consumed by same consumer in order.

However if you are saying all the messages are relevant and depend on each other, nothing much to do else than handling the concurrency yourself in the consumer side or better having only one partition and one consumer.

0

If you are writing to a topic T1 with 3 partitions, and your consumers are in a consumer group, then each consumer will consume from one partition of T1. Let's say C1 reads from Partition 1 and C2 reads from Partition 2 etc. There can be no guarantee on the ordering of data across multiple partitions, ordering is only guaranteed within a partition. Consider the following example:

P1 is producing the following records in order:

╔════════╗
║ Record ║
╠════════╣
║ R1     ║
║ R2     ║
║ R3     ║
║ R4     ║
║ R5     ║
║ R6     ║
╚════════╝

Let's say the records get partitioned in the following way:

╔═════════════╦═════════════╦═════════════╗
║ Partition 1 ║ Partition 2 ║ Partition 3 ║
╠═════════════╬═════════════╬═════════════╣
║ R1          ║ R2          ║ R3          ║
║ R4          ║ R5          ║ R6          ║
╚═════════════╩═════════════╩═════════════╝

Now C1 could finish reading all it's records before C2 and C3 even start. So record R4 will be processed before R2 and R3. However, the ordering in Partition 1 will always be preserved, so R1 will always be received and processed before R4.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.