When we submit application to Spark, and after performing any operation Spark Web UI displays Job and Stages like count at MyJob.scala:15
. But in my application there are multiple count
and save
operations are there. So it is very difficult to understand UI. Instead of count at MyJob.scala:15
, can we add custom description to give more detailed information to job.
While googling found https://issues.apache.org/jira/browse/SPARK-3468 and https://github.com/apache/spark/pull/2342, author attached image, with detailed description like 'Count', 'Cache and Count', 'Job with delays'. So can we achieve same? I am using Spark 2.0.0.