15

What are the disadvantages of mapreduce? There are lots of advantages of mapreduce. But I would like to know the disadvantages of mapreduce too.

5 Answers 5

27

I would rather ask when mapreduce is not a suitable choice? I don't think you would see any disadvantage if you are using it as intended. Having said that, there are certain cases where mapreduce is not a suitable choice :

  • Real-time processing.
  • It's not always very easy to implement each and everything as a MR program.
  • When your intermediate processes need to talk to each other(jobs run in isolation).
  • When your processing requires lot of data to be shuffled over the network.
  • When you need to handle streaming data. MR is best suited to batch process huge amounts of data which you already have with you.
  • When you can get the desired result with a standalone system. It's obviously less painful to configure and manage a standalone system as compared to a distributed system.
  • When you have OLTP needs. MR is not suitable for a large number of short on-line transactions.

There might be several other cases. But the important thing here is how well are you using it. For example, you can't expect a MR job to give you the result in a couple of ms. You can't count it as its disadvantage either. It's just that you are using it at the wrong place. And it holds true for any technology, IMHO. Long story short, think well before you act.

If you still want, you can take the above points as the disadvantages of mapreduce :)

HTH

6

Here are some usecases where MapReduce does not work very well.

  1. When you need a response fast. e.g. say < few seconds (Use stream processing, CEP etc instead)
  2. Processing graphs
  3. Complex algorithms e.g. some machine learning algorithms like SVM, and also see 13 drawfs (The Landscape of Parallel Computing Research: A View From Berkeley)
  4. Iterations - when you need to process data again and again. e.g. KMeans - use Spark
  5. When map phase generate too many keys. Thensorting takes for ever.
  6. Joining two large data sets with complex conditions (equal case can be handled via hashing etc)
  7. Stateful operations - e.g. evaluate a state machine Cascading tasks one after the other - using Hive, Big might help, but lot of overhead rereading and parsing data.
2
  1. You need to rethink/ rewrite trivial operations like Joins, Filter to achieve in map/reduce/Key/value patterns
  2. MapReduce assumes that the job can be parallelized. But it may not be the case for all data processing jobs.
  3. It is closely tied with Java, of course you have Pig and Hive for rescue but you lose flexibility.
1
  • You can always use streaming, if you don't want to use java.
    – Tariq
    Sep 3, 2013 at 21:50
1
  1. First of all, it streams the map output, if it is possible to keep it in memory this will be more efficient. I originally deployed my algorithm using MPI but when I scaled up some nodes started swapping, that's why I made the transition.

  2. The Namenode keeps track of the metadata of all files in your distributed file system. I am reading a hadoop book (Hadoop in action) and it mentioned that Yahoo estimated the metadata to be approximately 600 bytes per file. This implies if you have too many files your Namenode could experience problems.

  3. If you do not want to use the streaming API you have to write your program in the java language. I for example did a translation from C++. This has some side effects, for example Java has a large string overhead compared to C. Since my software is all about strings this is some sort of drawback.

To be honest I really had to think hard to find disadvantages. The problems mapreduce solved for me were way bigger than the problems it introduced. This list is definitely not complete, just a few first remarks. Obviously you have to keep in mind that it is geared towards Big Data, and that's where it will perform at its best. There are plenty of other distribution frameworks out there with their own characteristics.

3
  • NN has nothing to do with MR.
    – Tariq
    Sep 3, 2013 at 21:52
  • It's a limitation of the architecture running mapreduce, so it's something to keep in mind, although you are right, it need not be strictly MR related
    – DDW
    Sep 5, 2013 at 13:21
  • Mapreduce can run anywhere, not just HDFS. And NN is specific to HDFS. You'll see the metadata problem if you are storing a lot of very small files in your HDFS, which is again not the very efficient use of Hadoop platform. But, I agree. Whatever you said is also correct. The question was specific to the MR Framework, so I thought to mention that.
    – Tariq
    Sep 5, 2013 at 13:33
1

There are some disadvantages of MapReduce,

  • MapReduce is not suitable for real-time calculations and stream computing. It is designed for batch processing of large datasets, which can take a long time to complete.
  • MapReduce has a fixed processing pipeline and does not support iterations. It also lacks support for multi-input operators.
  • MapReduce is sensitive to configuration parameters and requires complicated tuning.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.