2

This is a question regarding the performance of writable variables and allocation within a map reduce step. Here is a reducer:

static public class MyReducer extends Reducer<Text, Text, Text, Text> {
      @Override
      protected void reduce(Text key, Iterable<Text> values, Context context) {
        for (Text val : values) {
            context.write(key, new Text(val));
        }
      }
}

Or is this better performance-wise:

static public class MyReducer extends Reducer<Text, Text, Text, Text> {
      private Text myText = new Text();
      @Override
      protected void reduce(Text key, Iterable<Text> values, Context context) {
        for (Text val : values) {
            myText.set(val);
            context.write(key, myText);
        }
      }
}

In the Hadoop Definitive Guide all the examples are in the first form but I'm not sure if that is for shorter code samples or because it's more idiomatic.

2 Answers 2

1

The book may use the first form because it is more concise. However, it is less efficient. For large input files, that approach will create a large number of objects. This excessive object creation would slow down your performance. Performance-wise, the second approach is preferable.

Some references that discuss this issue:

0

Yeah, second approach is preferable if reducer has large data to process. The first approach, will keep creating references and cleaning it up depends on the garbage collector.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.