1

Assume that I have a cluster of machines with different powers (i.e. different CPU, Memory, etc.). Is there any way to determine the location of a specific operator (or maybe task) to run? Is it possible to change the default running location of an operator?

I read about "Task chaining and resource groups". However, I think it is not solve my problem.

Is there any specific configuration in Flink? if no, What part of code may relates to this assumption?

2
  • This looks like a question for Server Fault
    – Sander
    Commented Aug 2, 2018 at 10:16
  • the best that Flink does is to allow you to use someStream.filter(...).slotSharingGroup("name"); as you might have seen at ci.apache.org/projects/flink/flink-docs-stable/dev/stream/…. And it is not a "must do" for the specific group. It only tries if it can. Because it follows the approach to deploy your job whenever is possible....
    – Felipe
    Commented Dec 15, 2020 at 16:28

2 Answers 2

1

The taskmanager.numberOfTaskSlots property can be an avenue to indicating the relative working capacity of different taskmanagers. You would likely have to adjust both the taskSlots and the memory allocation options based on the resources available resources on the machine.

Personally though, I prefer to address this type of problem with containers, like docker. That way, you can have all of your taskmanagers configure the same, and let the orchestration tools in your container system worry about assigning more or fewer virtual instances to a physical machine base on available resources.

1
  • 1
    Of-course I think using containers is good solution. However, for some purposes I want to manege my resources manually. I want to manually choose an operator with specific properties run on a specific machine. I don't know it is in the responsibility of Job manager or Resource manager. But I want to know who decide which operator is run on which machine (or task manger)? @CantankerousBullMoose
    – Folani
    Commented Aug 9, 2018 at 9:44
0

Each bunch of task can be tagged to some name and id as below:

inputStream
 .map()
 .keyBy()
 .window()
 .reduce()
 .name("enriching customer orders info")
 .uid("enriching customer orders info")

this way each operator can be identified uniquely you can see these names in DAG

hope this helps.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.