12

very simple questions: in which cases should I prefer Hadoop MapReduce over Spark? (I hope this question has not been asked yet - at least I didn't find it...)

I am currently doing a comparison of those two processing frameworks and from what I have read so far, everybody seems to suggest to use Spark. Does that also conform to your experience? Or can you name use cases where MapReduce performes better then Spark?

Would I need more ressources (esp. RAM) for the same task with Spark then I would need for MapReduce?

Thanks and regards!

0

5 Answers 5

5

Spark is a great improvement over traditional MapReduce.

When would you use MapReduce over Spark?

When you have a legacy program written in the MapReduce paradigm that is so complex that you do not want to reprogram it. Also if your problem is not about analyzing data then Spark might not be right for you. One example I can think of is for web crawling, there is a great Apache project called Apache Nutch, that is built on Hadoop not Spark.

When would I use Spark over MapReduce?

Ever since 2012... Ever since I started using Spark I haven't wanted to go back. It has also been a great motivation to expand my knowledge beyond Java and to learn Scala. A lot of the operations in Spark take less characters to complete. Also, using Scala/REPL is so much better to rapidly produce code. Hadoop has Pig, but then you have to learn "Pig Latin", which will never be useful anywhere else...

If you want to use Python Libs in your data analysis, I find it easier to get Python working with Spark, and MapReduce. I also REALLY like using something like IPython Notebook. As much as Spark learned me to learn Scala when I started, using IPython Notebook with Spark motivated me to learn PySpark. It doesn't have all the functionality, but most of it can be made up for with Python packages.

Spark also now features Spark SQL, which is backwardly compatible with Hive. This lets you use Spark, to run close to SQL queries. I think this is much better then trying to learn HiveQL, which is different enough that everything is specific to it. With Spark SQL, you can usually get away with using general SQL advice to solve issues.

Lastly, Spark also has MLLib, for machine learning, which is a great improvement over Apache Mahout.

Largest Spark issue: the internet is not full of troubleshooting tips. Since Spark is new, the documentation about issues is a little lacking... It's a good idea to buddy up with someone from AmpLabs/Databricks (the creators of Spark from UC Berkeley, and their consulting business), and utilize their forums for support.

4

You should prefer Hadoop Map Reduce over Spark if

  1. You have to query historic data, which in huge volumes of tera bytes/peta bytes in a huge cluster.
  2. You are not bothered about the job completion time - Job completion time in hours Vs minutes is not important to you
  3. Hadoop MapReduce is meant for data that does not fit in the memory whereas Apache Spark has a better performance for the data that fits in the memory, particularly on dedicated clusters.
  4. Hadoop MapReduce can be an economical option because of Hadoop as a service offering(HaaS) and availability of more personnel
  5. Apache Spark and Hadoop MapReduce both are failure tolerant but comparatively Hadoop MapReduce is more failure tolerant than Spark.

On other front, Spark’s major use cases over Hadoop

  1. Iterative Algorithms in Machine Learning
  2. Interactive Data Mining and Data Processing
  3. Spark is a fully Apache Hive-compatible data warehousing system that can run 100x faster than Hive.
  4. Stream processing: Log processing and Fraud detection in live streams for alerts, aggregates and analysis
  5. Sensor data processing: Where data is fetched and joined from multiple sources

Have a look at this blog and dezyre blog

0

Everyone is correct! Spark! Spark will not require more resources - you can tune it to take less RAM by making the RDD persistence to disk.

If planning to use Spark use 1.3 as the minimum version as there have been significant improvements.

Only scenario where you might end up using MR is if you have legacy code you want to continue with.

Another point to note is that people had moved away from writing MR code for quite sometime. There have been abstractions like Pig, Hive etc. on top of MR. Now all those abstractions will start supporting MR in future. And that is one of the places where Spark is still a bit lagging.

0

At the core, MR is a parallel programming model, and by its own right is superlative stuff. It has changed idea about how data can be thought and used.

Hadoop brought MR within reach of everybody with its design of distribute computing engine based on MR (and distributed file system, HDFS).

Now with these things are solved, more new needs started cropping up, and Hadoop MR (and any high level abstraction sitting on top of it) failed to deliver in 3 specific areas: a) Iterative computing b) Real time data processing c) Interactive usage

To solve these problems, Spark brought 2 important changes: 1. Generic DAG 2. Distributed data sharing

So, in essence, if you fall into these 3 use cases, most likely Spark will be preferable. If you are not, you may not get any additional benefit by using Spark (other than comfort of python and probably a "geek" nametag). In fact, in a smaller cluster or clusters with smaller config may perform better in Hadoop MR.

-1

As far as I know in Spark the whole single partition must fit into memory (2 GB is the maximum size because it uses ByteBuffer internally). That might be a problem when doing groupBy operation and one group is larger than this limit (each group is represented as a single Tuple2 item which cannot be partitioned).

So in some cases Spark may fail in comparison with MapReduce.

Link to the related issue in Spark JIRA https://issues.apache.org/jira/browse/SPARK-1476

3
  • Whole Partition does not HAVE to fit into memory. And 2 GB a node is NOT the Max. But I do find I usually only utilize about half the memory on my machine for Spark, (and on Azure this is usually about 16gig a node). May 18, 2015 at 12:39
  • I added link to the related issue in Spark JIRA. There is a 2GB limit.
    – vanekjar
    May 18, 2015 at 12:46
  • Blocks and Partitions are not the same thing. May 18, 2015 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.