I wrote a Spark job in Java. The job is packaged as a shaded jar and executed:
spark-submit my-jar.jar
In the code, there are some files (Freemarker templates) that reside in src/main/resources/templates
. When run locally, I'm able access the files:
File[] files = new File("src/main/resources/templates/").listFiles();
When the job is run on a cluster, a null-pointer exception is returned when the previous line is executed.
If I run jar tf my-jar.jar
I can see that the files are packaged in a templates/
folder:
[...]
templates/
templates/my_template.ftl
[...]
I'm just unable to read them; I suspect that .listFiles()
tries to access the local filesystem on the cluster node, and the files aren't there.
I'm curious to know how I should package files to be used within a self-contained Spark job. I'd rather not copy them to HDFS outside of the job because it becomes messy to maintain.