81

I am new to Apache Spark, and I just learned that Spark supports three types of cluster:

  • Standalone - meaning Spark will manage its own cluster
  • YARN - using Hadoop's YARN resource manager
  • Mesos - Apache's dedicated resource manager project

I think I should try Standalone first. In the future, I need to build a large cluster (hundreds of instances).

Which cluster type should I choose?

1
  • 1
    Note: Apache Mesos support is deprecated as of Apache Spark 3.2.0 (Oct 2021). It will be removed in a future version. (link). There is also an option to use Kubernetes as a cluster manager (link).
    – Niko Fohr
    Aug 11, 2022 at 13:55

4 Answers 4

76

I think the best to answer that are those who work on Spark. So, from Learning Spark

Start with a standalone cluster if this is a new deployment. Standalone mode is the easiest to set up and will provide almost all the same features as the other cluster managers if you are only running Spark.

If you would like to run Spark alongside other applications, or to use richer resource scheduling capabilities (e.g. queues), both YARN and Mesos provide these features. Of these, YARN will likely be preinstalled in many Hadoop distributions.

One advantage of Mesos over both YARN and standalone mode is its fine-grained sharing option, which lets interactive applications such as the Spark shell scale down their CPU allocation between commands. This makes it attractive in environments where multiple users are running interactive shells.

In all cases, it is best to run Spark on the same nodes as HDFS for fast access to storage. You can install Mesos or the standalone cluster manager on the same nodes manually, or most Hadoop distributions already install YARN and HDFS together.

2
  • 12
    "One advantage of Mesos over both YARN and standalone mode is its fine-grained sharing option": fine-grained sharing is not supported anymore as of Apache Spark 2.0.0
    – jgp
    Jul 29, 2016 at 12:50
  • Well written, just one more comment the standalone cluster only supports client mode for python application.
    – Daisy QL
    Jan 2, 2020 at 15:03
9

Standalone is pretty clear as other mentioned it should be used only when you have spark only workload.

Between yarn and mesos, One thing to consider is the fact that unlike mapreduce, spark job grabs executors and hold it for entire lifetime of a job. where in mapreduce a job can get and release mappers and reducers over lifetime.

if you have long running spark jobs which during the lifetime of a job doesn't fully utilize all the resources it got in beginning, you may want to share those resources to other app and that you can only do either via Mesos or Spark dynamic scheduling. https://spark.apache.org/docs/2.0.2/job-scheduling.html#scheduling-across-applications So with yarn, only way have dynamic allocation for spark is by using spark provided dynamic allocation. Yarn won't interfere in that while Mesos will. Again this whole point is only important if you have a long running spark application and you would like to scale it up and down dynamically.

-1

In this case and similar dilemmas in data engineering, there are many side questions to be answered before choosing one distribution method over another. For example, if you are not running your processing engine on more than 3 nodes, you usually are not facing too big of a problem to handle so your margin of performance tuning between YARN and SparkStandalone (based on experience) will not clarify your decision. Because usually you will try to make your pipeline simple, specially when your services are not self-managed by cloud and bugs and failures happen often.

I choose standalone for relatively small or not-complex pipelines but if I'm feeling alright and have a Hadoop cluster already in place, I prefer to take advantage of all the extra configs that Hadoop(Yarn) can give me.

-2

Mesos has more sophisticated scheduling design, allowing applications like Spark to negotiate with it. It's more suitable for the diversity of applications today. I found this site really insightful:

https://www.oreilly.com/ideas/a-tale-of-two-clusters-mesos-and-yarn

"... YARN is optimized for scheduling Hadoop jobs, which are historically (and still typically) batch jobs with long run times. This means that YARN was not designed for long-running services, nor for short-lived interactive queries (like small and fast Spark jobs), and while it’s possible to have it schedule other kinds of workloads, this is not an ideal model. The resource demands, execution model, and architectural demands of MapReduce are very different from those of long-running services, such as web servers or SOA applications, or real-time workloads like those of Spark or Storm..."

1
  • 2
    This (2/2015) refers to early releases of Hadoop 2.x and is totally oudated now as YARN schedulling has changed a lot over the last three years, with subsecuent versions of Hadoop 2.x and 3.x, to target this among other problems and current YARN design has almost nothing to do with early MapReduce architecture. – Oct 24, 2019 at 18:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.