12

I wrote a Spark job in Java. The job is packaged as a shaded jar and executed:

spark-submit my-jar.jar

In the code, there are some files (Freemarker templates) that reside in src/main/resources/templates. When run locally, I'm able access the files:

File[] files = new File("src/main/resources/templates/").listFiles();

When the job is run on a cluster, a null-pointer exception is returned when the previous line is executed.

If I run jar tf my-jar.jar I can see that the files are packaged in a templates/ folder:

 [...]
 templates/
 templates/my_template.ftl
 [...]

I'm just unable to read them; I suspect that .listFiles() tries to access the local filesystem on the cluster node, and the files aren't there.

I'm curious to know how I should package files to be used within a self-contained Spark job. I'd rather not copy them to HDFS outside of the job because it becomes messy to maintain.

1
  • Could you tell what is your master? and which deploy mode you are using? Apr 17, 2016 at 19:37

3 Answers 3

13

Your existing code is referencing them as files which are not packaged up and shipped to the Spark nodes. But, since they're inside your jar file you should be able to reference them via Foo.getClass().getResourceAsStream("/templates/my_template_ftl"). More info on Java resource streams here: http://www.javaworld.com/article/2077352/java-se/smartly-load-your-properties.html

11

It appears that running Scala (2.11) code on Spark does not support accessing resources in shaded jars.

Executing this code:

var path = getClass.getResource(fileName)
println("#### Resource: " + path.getPath())

prints the expected string when run outside of Spark.

When run inside Spark, a java.lang.NullPointerException is raised because path is null.

1
  • 2
    getResourceAsStream() seems to work for me, but not getResource().
    – Ted
    Oct 24, 2017 at 1:31
4

I have accessed my resource file like below in spark-scala. I've share my code please check.

val fs=this.getClass().getClassLoader().getResourceAsStream("smoke_test/loadhadoop.txt")

val dataString=scala.io.Source.fromInputStream(fs).mkString

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.