1

This feels obvious, but I'm asking anyway since I can't find a clear confirmation in the documentation:

The semantics of the Flink Table API upsert kafka connector available in Flink 1.12 match pretty well the semantics of a Kafka compacted topics: interpreting the stream as a changelog and using NULL values as tombstone to mark deletions.

So my assumption is that it is ok to use it to consume from and produce to a compacted topic, and it's probably made precisely for that, although it should work fine as well with a non-compacted topic assuming its content is indeed a changelog. But I'm surprised not to find any reference to compacted topic in that part of the documentation.

Could somebody please confirm or infirm this assumption?

1 Answer 1

1

Yes, it was made for use with compacted topics. According to FLIP-149:

Generally speaking, the underlying topic of the upsert-kafka source must be compacted. Besides, the underlying topic must have all the data with the same key in the same partition, otherwise, the result will be wrong.

5
  • Thanks. There were indeed more information in the FLIP, including the compacted topics and behavior of UPDATE_BEFORE events. All examples I see are only showing the sink side of the connector unfortunately, although the doc also describes the mechanism of the source to be what we expect, I think I get it now.
    – Svend
    Jan 9, 2021 at 21:02
  • 2
    You can find an example using kafka-upsert as a source connector in this repository.
    – morsapaes
    Jan 11, 2021 at 9:21
  • Can you use regular kafka debezium avro format with compaction? Upsert Kafka doesn't seem to support debezium from what the docs say.
    – Awesome-o
    Feb 26, 2021 at 23:48
  • **Specifically as a source in my case.
    – Awesome-o
    Feb 27, 2021 at 0:41
  • @david-anderson The doc says: Generally speaking, the underlying topic of the upsert-kafka source must be compacted, but in my test, non-compacted also works fine for me, not sure I have missed something..
    – Tom
    Jul 1, 2021 at 1:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.