I am using Flink v.1.4.0
.
I want to be able to name the operators in the Flink UI
. I understand that to do so all I need is to just use the .name()
method on a DataSet
or DataStream
. For example, like this:
DataSet<SomePojo> flinkDataSet = ...;
flinkDataSet
.map(new SomeTransformation())
.name("Transformation A");
What I would like to know is what exactly defines an operator? Is it the DataSet's/DataStream's key? or maybe the element types? I understand that the operator will remain the same if I apply a MapFunction
that has the same return type as the element the function was applied on, but noticed that the operator usually changes if I apply a FlatMapFunction
or a KeyBy
or GroupBy
operation, as well as applying actions of, course.
The reason I am asking this is because I want to understand how granular I can get the Flink DAG to be in the Flink UI, e.g. could I name every transformation? or are operators defined by the type of the DataSet/DataStream being processed and hence I can only name those? For example, would this work?
flinkDataSet
.map(new SomeTransformation())
.name("Transformation A")
.map(new SomeOtherTransformation())
.name("Transformation B");
or would the first name be overwritten by the second? I can experiment and find out of course, but I would like to understand the whys too.