7

Does the MapReduce abstraction a good one for dealing with problems even in a single machine? For example, I have a 12-core machine and I have to count words in thousands of files (classic MapReduce example).

Using a MapReduce implementation with Mappers and Reducers in multiple threads is a good way to solve this problem, considering that we're working on a single machine with a single Hard-drive?

I guess my question comes down to this: Is the MapReduce paradigm good only for working in a cluster of machines?

4
  • I'd wager the "single disk" is the big point. I expect a single core could run isspace(3) on chars coming from a spinning hard disk as fast as the drive can serve them. Will any of the files already be in the buffer cache because they are frequently used?
    – sarnold
    Jun 24, 2011 at 20:17
  • To make it more interesting, then, we can say that we have a large buffer of files in memory. Let's say we have 16GB of RAM to hold files and the MapReduce job consumes this buffer. And if there is no Disk latency? If all data is in RAM? Jun 24, 2011 at 20:21
  • In that case, I'd expect multiple threads of computation to be worthwhile.
    – sarnold
    Jun 24, 2011 at 20:23
  • That's still not going to help - you're still going to be blocking on disk bandwidth reading the file in in the first place. Jun 27, 2011 at 1:33

2 Answers 2

9

In general you can have two situations:

  1. Your problem is small enough to fit into the memory of your single system and your single system has enough CPU power to solve the problem within the required time.
  2. Your problem is too big. 2.1 Running time is too big (disk IO and/or CPU time) 2.2 Too big to fit into memory (RAM).

For 2.1 and 2.2 the MapReduce paradigm helps a lot in splitting the work into many smaller chunks. If you need more CPU you simply add CPUs.

So if you have a single system and it turns out your problem is too big to fit into memory (point 2.2) you can still benefit from the fact that MapReduce can easily put a part of the problem on disk until that part is to be processed.

The important fact is that if you have a problem that is small enough to fit into memory and small enough to be processed on a single system then a dedicated (non-MapReduce) solution can be a lot faster.

2

I guess my question comes down to this: Is the MapReduce paradigm good only for working in a cluster of machines?

Generally, yes: MapReduce is likely to be less efficient on a single PC. I can't think of many (if any) situation that MapReduce would have an advantage over more resource-optimized approaches when used in a non-distributed environment (i.e. single PC, single hard drive). In other words, if you are trying to squeeze every little bit of performance out of your single PC, you will most likely be able to achieve it with a custom solution instead of MapReduce.

However, if you plan on adding more nodes and creating a cluster, then MapReduce will be the go-to paradigm.

1
  • If the work done by each map task was more substantial (eg, computationally intensive), a mapreduce approach could be a good choice for a single machine. Jun 27, 2011 at 1:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.