4

I just want to verify my understanding about these parameters and their relationship, if I am wrong then please notify me.

  • mapreduce.reduce.shuffle.input.buffer.percent tells total amount of memory allocated to entire shuffle phase of a reducer.

  • mapreduce.reduce.shuffle.memory.limit.percent tell maximum percentage of the in-memory limit that a single shuffle can consume from mapreduce.reduce.shuffle.input.buffer.percent.

  • mapreduce.reduce.shuffle.merge.percent is the usage threshold at which an in-memory merge will be initiated, expressed as a percentage of the total memory( mapreduce.reduce.shuffle.input.buffer.percent) allocated for storing in-memory map outputs.

  • But Hadoop-2.6 has a restriction that mapreduce.reduce.shuffle.merge.percent should be greater than mapreduce.reduce.shuffle.memory.limit.percent. It means a single shuffle has all the keys of same type otherwise what is the purpose of this restriction and what is the relation between all three ?

1 Answer 1

2

I would like to share my understanding on these properties, hope it would help. Advise me if anything is wrong.

mapreduce.reduce.shuffle.input.buffer.percent tells about the percentage of the reducer's heap memory to be allocated for the circular buffer to store the intermediate outputs copied from multiple mappers.

mapreduce.reduce.shuffle.memory.limit.percent tells about the maximum percentage of the above memory buffer that a single shuffle (output copied from single Map task) should take. The shuffle's size above this size will not be copied to the memory buffer, instead they will be directly written to the disk of the reducer.

mapreduce.reduce.shuffle.merge.percent tells about the threshold percentage by where the in-memory merger thread will run to merge the available shuffle contents on the memory buffer into a single file and immediately spills the merged file into the disk.

It is obvious that in-memory merger thread should require at least 2 shuffle files to be present in memory buffer to initiate merge. So at any time the mapreduce.reduce.shuffle.merge.percent should be higher than any single shuffle file in memory controlled by mapreduce.reduce.shuffle.memory.limit.percent property, by which it mandates that there should be at least more than one shuffle file to be present in the buffer for the merge process.

3
  • @Priyank, is the answer helpful to you get clarification?, if not let me know.
    – suresiva
    Apr 30, 2015 at 13:41
  • @ hserus, your answer is helpful but i do not know how to verify it.
    – Novice
    May 3, 2015 at 11:02
  • @priyank, if you set the mapreduce.reduce.shuffle.merge.percent value below than mapreduce.reduce.shuffle.memory.limit.percent, then you will get the exception on reducer phase. The explanation of the scenario is given in the answer..let me know what else clarification u need?
    – suresiva
    May 3, 2015 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.