2

In the case of Scala, we can use SparkContext.applicationId but I was not able to find any way to do the same using Spark's java API.

Using SparkConf.getAppId() results in NoSuchElementException: spark.app.id

Is there any other unique Id that can be used?

My use case: I want the Spark job to not fail in case output from the previous job already exist and want to avoid deleting the output of previous job and avoid using random UUID

2 Answers 2

4

You can use SparkContext.applicationId:

JavaSparkContext jsc = ...;
jsc.sc.applicationId;
2
  • Thanks, is there any way to get the retry count as well? I guess applicationId will be same in case of retry
    – user401445
    Commented Dec 12, 2016 at 21:01
  • 1
    When you have application id you can use REST API (spark.apache.org/docs/latest/monitoring.html#rest-api) but I don't know if there is any other way.
    – user6022341
    Commented Dec 12, 2016 at 21:03
1

Java code

System.out.println("appid = " + spark.sparkContext().applicationId());

Result

appid = app-20171023065927-0026

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.