8

I am new with Spark and I wanted to ask some common guidelines about developing and testing my code for Apache Spark framework

  1. What is the most common setup to test my code locally? Is there any built VM to raise (ready box etc.)? Do I have to setup locally spark? Is there any test library to test my code?

  2. When going in cluster mode I notice that there are some ways to setup your cluster; production wise, what is the most common way to setup a cluster to run Spark? Three options here

    • Standalone cluster setup
    • With YARN
    • With MESOS

Thank you

1

4 Answers 4

7

1) Common setup: Just download the Spark version on a local machine. Unzip it and follow these steps to set it up locally.

2) Launching a cluster for production: The Spark cluster mode overview available here explains the key concepts when running a Spark cluster. Spark can be run both in a standalone way and on several existing cluster managers. Currently, several deployments options are available:

  • Amazon EC2

  • Standalone mode

  • Apache Mesos

  • Hadoop YARN

EC2 scripts let you launch a cluster in about 5 minutes. In fact, if you are using EC2, the best way to go is using the script provided by spark. The standalone mode is the best for the deployment of Spark on a private cluster.

Normally, we use YARN as cluster manager when we have an existing Hadoop setup with YARN, and the same goes for Mesos. Instead, if you are creating a new cluster from the dust, I would recommend using the Standalone mode, considering you are not using Amazon's EC2 instances. This link shows some steps that help arranging a Standalone Spark cluster.

6
  • Hi thanks for your reply, question: do I also need a hadoop installation to start spark? for example the prebuild for hadoop 2.4: d3kbcqa49mib13.cloudfront.net/spark-1.1.0-bin-hadoop2.4.tgz
    – tbo
    Oct 8, 2014 at 11:24
  • 1
    You don't need. Just download the above tar and unzip it and follow steps. you can run simple programs. its not necessary to have hadoop installed. if you want to read files form HDFS to process using spar then you need HDFS setup. Oct 8, 2014 at 11:37
  • Therefore, the prebuild for hadoop is in the case I need to use hadoop version 2.4, Ok. Why there is no standalone prebuild? Or am I missing something? What is the benefit of prebuilding for hadoop, can't just in my code logic interact with hadoop and take the data I need? I'm trying to understand it better... Thanks again
    – tbo
    Oct 8, 2014 at 11:43
  • The hadoop pre build version has , hadoop core library included so that hadoop API can be accessed.e.g saveashadoopfile.it just has library included so that hadoop functionalities can be accessed. Oct 8, 2014 at 12:14
  • 1
    Take a look here (spark.apache.org/faq.html). As stated by @SandeshDeshmane, you will need some kind of distributed file system in order to interact with file partitioning. The HDFS provided by Hadoop is an adequate solution for Spark, and that is why many people uses Hadoop along with Spark. Spark does not depend on Hadoop to work, but again, it needs some kind of DFS like the one used by Hadoop (HDFS). Oct 8, 2014 at 14:25
3

Sandbox from Hortonworks hope will help.

HDP 2.2.4 Sandbox with Apache Spark & Ambari Views http://hortonworks.com/products/hortonworks-sandbox/#install

Second resource I'm using is http://www.cloudera.com/downloads/quickstart_vms/5-8.html

The image does contain Hadoop, HBase, Impala, Spark and many more features. Does require 4gb RAM, 1 CPU and 62.5GB disk. Kind large but is free and does fulfill all requirements rather paid versions on cloud.

0
1

I am using Sandbox from Hortonworks. It has hadoop, java and all the required environment to run the spark examples.

I would suggest you to write simple Java or Scala class in your IDE. Create SparkConf and SparkContext objects in your "SimpleApp.java".

SparkConf conf = new SparkConf().setAppName(appName).setMaster("local[2]");
JavaSparkContext sc = new JavaSparkContext(conf);

Once you run maven clean package or maven package it will create jar file in your project's target folder. If it doesn't then create a JAR file using following command. You can find the SimpleApp.class file in "target/classes" folder. cd to this directory.

jar cfve file.jar SimpleApp.class

Put this JAR file into your project in target directory. This JAR file contains the dependency of your SimpleApp class while submitting your job to Spark. I guess you have project structure like below.

simpleapp
 - src/main/java
  - org.apache.spark.examples
    -SimpleApp.java
 - lib
  - dependent.jars (you can put all dependent jars inside lib directory)
 - target
  - simpleapp.jar (after compiling your source)

cd to your spark directory. I am using spark-1.4.0-bin-hadoop2.6. Your cmd looks like this.

spark-1.4.0-bin-hadoop2.6>

Start the master and worker using following commands.

spark-1.4.0-bin-hadoop2.6> ./sbin/start-all.sh

If this does not work then start master and slaves separately.

spark-1.4.0-bin-hadoop2.6> ./sbin/start-master.sh
spark-1.4.0-bin-hadoop2.6> ./sbin/start-slaves.sh

Submit your spark program using Spark Submit. If you have structure like I explained then pass this argument in class.

--class org.apache.spark.examples.SimpleApp

else

--class SimpleApp

Finally submit your spark program through spark submit.

spark-1.4.0-bin-hadoop2.6>./bin/spark-submit --class SimpleApp --master local[2] /PATH-TO-YOUR-PROJECT-DIRECTORY/target/file.jar

Here I have used local[2] as a master so my program will run on two threads but you can pass your master URL in --master as a --master spark://YOUR-HOSTNAME:7077

Port number 7077 is a default port number for Master URL.

1

For development and testing, I recommend using sbt in local mode so that you can avail yourself of the test suites you will write for your code. The following advice is verbatim from Matei Zaharia regarding Spark development, and it's always worked for me:

"As a tip (and maybe this isn't documented well), I normally use SBT for development to avoid the slow build process, and use its interactive console to run only specific tests. The nice advantage is that SBT can keep the Scala compiler loaded and JITed across builds, making it faster to iterate. To use it, you can do the following:

  • Start the SBT interactive console with sbt/sbt
  • Build your assembly by running the "assembly" target in the assembly project: assembly/assembly
  • Run all the tests in one module: core/test
  • Run a specific suite: core/test-only org.apache.spark.rdd.RDDSuite (this also supports tab completion)"

Source: http://mail-archives.apache.org/mod_mbox/spark-dev/201412.mbox/%3cCAAsvFP=V9FL=KvXNUeWVfFz8q04oj1exdwSaSNiab5Vc9hFkUg@mail.gmail.com%3e

1
  • (This is from the context of development of Spark source code, but the same principle applies for your own scala projects)
    – modulus0
    Oct 17, 2015 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.