24

When we submit application to Spark, and after performing any operation Spark Web UI displays Job and Stages like count at MyJob.scala:15. But in my application there are multiple count and save operations are there. So it is very difficult to understand UI. Instead of count at MyJob.scala:15, can we add custom description to give more detailed information to job.

While googling found https://issues.apache.org/jira/browse/SPARK-3468 and https://github.com/apache/spark/pull/2342, author attached image, with detailed description like 'Count', 'Cache and Count', 'Job with delays'. So can we achieve same? I am using Spark 2.0.0.

2 Answers 2

41

use the sc.setJobGroup:

Examples:
python:

In [28]: sc.setJobGroup("my job group id", "job description goes here")
In [29]: lines = sc.parallelize([1,2,3,4])
In [30]: lines.count()
Out[30]: 4

Scala:

scala> sc.setJobGroup("my job group id", "job description goes here")
scala> val lines = sc.parallelize(List(1,2,3,4))
scala> lines.count()
res3: Long = 4

SparkUI:

job description screen shot

I hope this is what you are looking for.

1
  • 1
    Note, if you happen to be using an org.apache.spark.sql.SparkSession (which was added later to unify all the different sql, spark, and hive contexts), then just use yourSparkSession.sparkContext to set this in the SparkContext. Aug 18, 2021 at 21:48
2

Note that new Zeppelin 0.8 loose his tracking hook if you change the JobGroup name, and can't display his job progress bar (job still working, no effect on the job itself)

You can use

sc.setLocalProperty("callSite.short","my job description")
sc.setLocalProperty("callSite.long","my job details long description")

instead

See How to change job/stage description in web UI? for some screen captures and scala syntax

2
  • 1
    Thanks for the answer! Does your answer also mean that we should prefer c.setLocalProperty over sc.setJobGroup? If so, I will point future readers to your answer. I am currently not working on apache-spark, if you are, you might tell that :)
    – Ajeet Shah
    May 24, 2020 at 20:13
  • How to enable the same for %sql paragraphs in Zeppelin?
    – mrbrahman
    Feb 19, 2021 at 22:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.