4

I want a formula of course Heuristic , but which can be used as a rule of thumb for deciding the following

  1. Increase partitions to X partitions for optimal performance and best utilisation of the cluster resources.

  2. Decrease partitions to X partitions for optimal performance and best utilisation of the cluster resources.

  3. Don't repartition the data , default partition are good enough for optimal performance and best utilisation of cluster resources.

I have this information

  1. No of nodes
  2. No of cores
  3. No of executors
  4. Memory available per executor
  5. Size of the Data
  6. Total memory etc

What could be a rule of thumb , a formula to calculate the no of partitions are run time for best performance.

Say If I have rdd and its default partitions are "X" then after applying the formula using the above info I should be able to decide "y" by either increasing , decreasing or not repartitoning.

Of course there cannot be an exact formula for this , but definitely a close approach would also just be fine.

Thanks

1

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.