0

The intermediate key-value pairs in a mapreduce job are written to mapred.local.dir before being shuffled to the tasktracker node which will run the reduce task.

I know HFDS is optimized to write large Blocks of data therefore minimizing the seek-time of a hard disk as compared to a regular filesystem.

Now I was curious if hadoop is optimized for streaming intermediate kev-value pairs to the local filesystem as well?

I am asking this because my application has little input data, but a huge amount of intermediate data and medium size output data. Is hadoop in my case beneficial or should I consider a different framework? (Note that my software is very closely related to WordCount, but I emit all substrings instead of all words)

Thanks a lot for any help!

EDIT: I reprased the question somewhat since at first glance I give the impression that intermediate kv pairs were sent to HDFS, they are sent to the local filesystem of the tasktracker node!

2 Answers 2

1

Is HDFS optimized for intermediate data?

Like @Tariq mentioned, HDFS is not used for intermediate data (though some people have successfully explored this idea).

So, let me rephrase your question:

Is Hadoop optimized for intermediate data?

Yes, there are some optimizations in place (for example, see the MAPREDUCE-3289 JIRA).

Even with these optimizations in place, shuffle-heavy jobs will see a bottleneck in this phase. Tuning the configuration parameters (like mapreduce.reduce.shuffle.input.buffer.percent) can help alleviate this problem to some extent. Using a combiner (as suggested by @Tariq) is also a good idea.

Is hadoop in my case beneficial or should I consider a different framework?

Yes, Hadoop is still useful in your case (assuming you are not running in single-node mode). You could do better writing your own code optimized for your particular use case, but that would be too much work (you'd have to deal with failures yourself, etc.) to justify doing it (in most cases).

2
  • 1
    I am aware that intermediate kv pairs are not written to HDFS, I was curious if the streaming was optimized for this phase since even in the terasort benchmark there is a lot of shuffling. Maybe the circular io.sort buffer is doing somewhat better than a regular output buffer? The reduce.input.buffer.. caused me some problems, since if I understand correctly it is part of the heap of the reduce task and I need all heap for the reduce() function, this caused my simulation to crash. But I agree that even if it's only for the failure handling it is definitely worth it.
    – DDW
    Nov 15, 2013 at 9:01
  • 1
    The paper was quite interesting, hopefully hadoop will consider implement an intermediate storage system, for my case it would be great since the amount of kv data is almost 10^4 times larger then the input.
    – DDW
    Nov 15, 2013 at 9:32
1

The intermediate output gets stored on the local FS and not on HDFS. So, it doesn't matter how much optimized HDFS is. But if you want to spread disk i/o to make things more efficient, you can use a comma-separated list of directories on different devices as the value for mapred.local.dir property. This will spread the load and thus increase the performance.

You could also make use of combiner to make things better.

1
  • Thx for the comment. I know it goes to local FS, just wondered if this streaming had some optimizatoins as well. Unfortunately in my case a combiner is not beneficial, there are too many different words, using a combiner would only decrease the output to something like 50-70%.
    – DDW
    Nov 15, 2013 at 8:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.