1

i'm currently working on a streaming ML pipeline and need exactly once event processing. I was interested by Flink but i'm wondering if there is any way to alter/update the execution state from outside.

The ml algorithm state is kept by Flink and that's ok, but considering that i'd like to change some execution parameters at runtime, i cannot find a viable solution. Basically an external webapp (in GO) is used to tune the parameters and changes should reflect in Flink for the subsequent events.

I thought about:

  • a shared Redis with pub/sub (as polling for each event would kill throughput)
  • writing a custom solution in Go :D
  • ...

The state would be kept by key, related to the source of one of the multiple event streams coming in from Kafka.

Thanks

1 Answer 1

1

You could use a CoMapFunction/CoFlatMapFunction to achieve what you described. One of the inputs is the normal data input and on the other input you receive state changing commands. This could be easiest ingested via a dedicated Kafka topic.

3
  • Looks promising, i was thinking of sending "control events" to Flink but as i haven't found anything on the internet i thought that i could just be abusing something that is not tailored to the purpose...i mean maybe Flink (and friends) are not the best choice for this application? Is my case so unusual?
    – Peterdeka
    Jul 21, 2016 at 14:30
  • I think the co functions are exactly made for these kind of use cases where you have multiple input streams but still want to access the same state where the elements arrive. Jul 21, 2016 at 15:03
  • Thanks Till, so i'll try them!
    – Peterdeka
    Jul 21, 2016 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.