4

I am planning to use for producing events to Kafka and I need strong delivery and ordering guarantees. By using enable.idempotence=true paramater in producer config I can have these guarantees.

My questions are:

  1. How can I achieve this with debezium?
  2. Is these guarantees provided by default?
  3. Is it configurable? How?

Note: I am planning to use Debezium Connector for PostgreSQL

1 Answer 1

7

Delevery Sematincs

guarantees at-least-once delivery:

When all systems are running nominally or when some or all of the systems are gracefully shut down, then consuming applications can expect to see every event exactly one time. However, when things go wrong it is always possible for consuming applications to see events at least once.

Source: Why must consuming applications expect duplicate events?


Ordering Guarantees

has strong ordering guarantees:

Most connectors will record all events for a single database table to a single topic. Additionally, all events within a topic are totally-ordered, meaning that the order of all of those events will be maintained. (Even if events are duplicated during failures, the end result after applying all of the events will remain the same.)

Source: How are events for a database organized?

3
  • Maybe, all events within a topic are totally-ordered within a single partition, so no total order across different partitions of the same topic, and thus if a consumer group is reading concurrently from diffrent partitions, the consumption order as comapred to the database log might not be maintained? am I correct/missing something? Commented Oct 2, 2020 at 15:21
  • exactly once, mean that no mater what it will never produce duplicate events. It's not the case , so Debezium is at least once
    – raphaelauv
    Commented Apr 6, 2022 at 9:25
  • 1
    @MazenEzzeddine You are absolutely right there. If we are using multiple partitions then the order is only guaranteed on per partition basis. To ensure that all the inserts, updates, and deletes for an entity go to the same Kafka partition one can use the message.key.columns setting in Debezium. Refer more - stackoverflow.com/questions/64932211/…
    – surajs21
    Commented Feb 17, 2023 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.