0

I've been trying to test a spark application on my local laptop before deploying it to a cluster (to avoid having to package and deploy my entire application every time) but struggling on loading the spark config file.

When I run my application on a cluster, I am usually providing a spark config file to the application (using spark-submit's --conf). This file has a lot of config options because this application interacts with Cassandra and HDFS. However, when I try to do the same on my local laptop, I'm not sure exactly how to load this config file. I know I can probably write a piece of code that takes the file path of the config file and just goes through and parses all the values and sets them in the config, but I'm just wondering if there are easier ways.

Current status:

  • I placed the desired config file in the my SPARK_HOME/conf directory and called it spark-defaults.conf ---> This didn't get applied, however this exact same file runs fine using spark-submit
  • For local mode, when I create the spark session, I'm setting Spark Master as "local[2]". I'm doing this when creating the spark session, so I'm wondering if it's possible to create this session with a specified config file.

2 Answers 2

0

Did you added --properties-file flag with spark-defaults.conf value in your IDE as an argument for JVM?

In official documentation (https://spark.apache.org/docs/latest/configuration.html) there is continuous reference to 'your default properties file'. Some options can not be set inside your application, because the JVM has already started. And since conf directory is read only through spark-submit, I suppose you have to explicitly load configuration file when running locally.

This problem has been discussed here: How to use spark-submit's --properties-file option to launch Spark application in IntelliJ IDEA?

2
  • Thanks for the response! What I'm not understanding fully is howcome when I launch spark shell with no parameters, spark-defaults.conf gets picked up with all the desired values. However, when I run it using a SparkSession.builder (i.e. from IDE or Code directly), then the spark-defaults.conf file is not being used. Is there a way to figure out what SPARK_HOME is being used when running from the IDE? I'm wondering if this has anything to do with it. Feb 4, 2019 at 21:44
  • I don't think it is a matter of path, because you are using jars from dependencies. Since we can specify parameters on different places during development, creators of Spark had to use some sort of precedence rules to address potential conflicts. Submitting application on distributed (or pseudo-distributed) cluster is simply a different situation, with different issues. Feb 5, 2019 at 6:51
0

Not sure if this will help anyone, but I ended up reading the conf file from a test resource directory and then setting all the values as system properties (copied this from Spark Source Code):

//_sparkConfs is just a map of (String,String) populated from reading the conf file
for {
  (k, v) ← _sparkConfs
} {
  System.setProperty(k, v)
}

This is essentially emulating the --properties-file option of spark-submit to a certain degree. By doing this, I was able to keep this logic in my test setup, and not need to modify the existing application code.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.