3

I am using Flink v.1.4.0.

I want to be able to name the operators in the Flink UI. I understand that to do so all I need is to just use the .name() method on a DataSet or DataStream. For example, like this:

DataSet<SomePojo> flinkDataSet = ...;

flinkDataSet
    .map(new SomeTransformation())
    .name("Transformation A");

What I would like to know is what exactly defines an operator? Is it the DataSet's/DataStream's key? or maybe the element types? I understand that the operator will remain the same if I apply a MapFunction that has the same return type as the element the function was applied on, but noticed that the operator usually changes if I apply a FlatMapFunction or a KeyBy or GroupBy operation, as well as applying actions of, course.

The reason I am asking this is because I want to understand how granular I can get the Flink DAG to be in the Flink UI, e.g. could I name every transformation? or are operators defined by the type of the DataSet/DataStream being processed and hence I can only name those? For example, would this work?

flinkDataSet
    .map(new SomeTransformation())
    .name("Transformation A")
    .map(new SomeOtherTransformation())
    .name("Transformation B");

or would the first name be overwritten by the second? I can experiment and find out of course, but I would like to understand the whys too.

2 Answers 2

4

As far as i have used the name() function, each tranformation that you can do using the datastream or dataset API is an operation, so in your second case the graph will look like this

"Transformation A" -------> "Transformation B"

But on the Flink dashborad, there's the posibility that those tasks could be grouped inside the same task like this:

-----------------------         -----------------------
| SOURCE              |  --->   |"Transformation A" ->| ---> More operations
-----------------------         |"Transformation B"   |
                                -----------------------
2

Yes, you can name every transformation. But as @diegoreico noted, if operators are chained*, then the Flink UI will show them grouped into a single box (node) in the DAG.

*chained operators: see Flink docu Task Chaining and Resource Groups

2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.