I wanted to enquire that which configuration option is given priority in spark? Is it the configuration file or the options we manually specify when running the spark-submit shell? What if I have different options for executor memory in my configuration file and I specify a different value while running the spark-submit shell?
1 Answer
The Spark (1.5.0) configuration page clearly states what the priorities are:
Properties set directly on the SparkConf take highest precedence, then flags passed to spark-submit or spark-shell, then options in the spark-defaults.conf file.
So this is the priority order (from highest to lowest):
- Properties set on the SparkConf (in program).
- Flags passed to
spark-submit
orspark-shell
. - Options set in the
spark-defaults.conf
file.