2

I am trying to set max memory for a Spark application running locally. I have tried many different ways.

In the program

SparkConf conf = new SparkConf();
conf.setMaster("local[2]");
conf.setAppName("app");     
conf.set("spark.executor.memory", "4G");
conf.set("spark.driver.memory", "4G");
sc = new JavaSparkContext(conf);

And when executing

./bin/spark-submit.cmd ./local/app.jar --master local[2] --driver-memory 4G

But I always get the following

INFO [MemoryStore] MemoryStore started with capacity 366.3 MB

and Spark spills RDD to disk. Spark version is 2.0.

What could I do?

2
  • 2
    did you try small 'g' ?
    – numX
    Sep 12, 2016 at 20:15
  • @numX I tried small g and the results are same.
    – Yumi F
    Sep 13, 2016 at 6:13

3 Answers 3

6

In local mode your executor is your driver and it is therefore a fraction of your heap. By the time spark-submit is executed the JVM has already been launched, so you need to pass in the --driver-memory arg. For example:

spark-submit --driver-memory 4g --class MyClass spark-app-1.0.jar

will start the JVM with 4G.

Order also seems to matter i.e. --driver-memory needs to come before --class

2
  • 2
    This helped me out! The order of the arguments is indeed important! Jun 12, 2017 at 9:02
  • 2
    This helped me! For PySpark, spark-submit --driver-memory 30g xxx.py works well, but spark-submit xxx.py --driver-memory 30g failed
    – Ghrua
    Nov 1, 2018 at 9:16
2

When running locally, spark will not used more memory then the java process has available. You should try running your application with extra memory vm params: -Xms2048m -Xmx4096m

1
  • This also does not work. But I change Spark version to 1.6.1 and it is OK now.
    – Yumi F
    Sep 13, 2016 at 12:32
0

The spark.driver.memory works for local running instance in Spark 2.

Spark forbids '-Xms' and converts spark.driver.memory into '-Xmx'.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.