0

I am currently using Kafka streaming DSL suppress feature for day window. we might have a situation where some of the events might come very late, beyond the grace period.

As per kafka streaming documentation such events will be discarded which does not fit into window.

Please help me out.

1) Is it possible to get hold of such discarded events in the same flow?

Apache flink does provide hold of such very late events and would like to know if such feature available in streaming.

2) How feasible to hold intermittent aggregated data in the memory with DSL- suppress for day window considering millions of events flow through system?

Any timeline kafka streaming community will provide rockDB support soon to avoid application crash due to out of memory.

1 Answer 1

1

I am currently using kafka streaming DSL suppress feature for day window. we might have situation where some of the events might come very late,beyond grace period.

As per kafka streaming documentation such events will be discarded which does not fit into window. [...]

1) Is it possible to get hold of such discarded events in the same flow?

You need to increase the grace period. The point of the grace period is to allow you to define for how long you may accept (very) late events to arrive. The grace period can actually be longer than the window size -- I mention this because you mentioned "which does not fit into window".

It seems to me as if you to accept late events, but you don't want to increase the grace period. Why?

Apache flink does provide hold of such very late events and would like to know if such feature available in streaming.

If you mean: Is there something like a callback for such very late events in Kafka Streams, then the answer is No, there is not.

2) How feasible to hold intermittent aggregated data in the memory with DSL- suppress for day window considering millions of events flow through system?

Any timeline kafka streaming community will provide rockDB support soon to avoid application crash due to out of memory.

For other readers: RocksDB is already supported and the default state store engine for all stateful operations in Kafka Streams. The only exception is the current implementation of the Supress() functionality, where the suppress buffer is not yet maintained via RocksDB.

Regarding your question: The work on KAFKA-7224: Add spill-to-disk for Suppression is in progress, but the exact ETA is not clear yet.

3
  • PART1: Planning for 2 types of use cases for stats calculation. 1) Realtime - Working as expected. 2) On demand delayed transaction which might be file feed with older dates, probably week old. Transaction feed --> INPUT TOPIC --> WINDOWED STREAMING APP -- > OUTPUT TOPIC --> CASSANDRA
    – Swapnil
    May 19, 2019 at 22:35
  • PART2: Dont want to keep grace period for longer duration as manual feed might come with week delay. Plan is to keep 1 day window with 15 mins grace period so that emits happens on time and stats available into Cassandra.
    – Swapnil
    May 19, 2019 at 22:37
  • PART3: If I get a mechanism to verify particular transaction(delayed feed) is rejected by current window during streaming processing then I can directly get existing stats from cassandra + calculate summary in the memory+ re add into Cassandra). OUTPUT Topic wont be applicable for history transaction stats. Main agenda is to use same realtime setup for history transaction feed. Hope I am able to clarify. Please guide.
    – Swapnil
    May 19, 2019 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.