I am trying to set max memory for a Spark application running locally. I have tried many different ways.
In the program
SparkConf conf = new SparkConf();
conf.setMaster("local[2]");
conf.setAppName("app");
conf.set("spark.executor.memory", "4G");
conf.set("spark.driver.memory", "4G");
sc = new JavaSparkContext(conf);
And when executing
./bin/spark-submit.cmd ./local/app.jar --master local[2] --driver-memory 4G
But I always get the following
INFO [MemoryStore] MemoryStore started with capacity 366.3 MB
and Spark spills RDD to disk. Spark version is 2.0.
What could I do?