Apache Spark has 3 blocks of memory:
- Cache - this is where RDDs are put when you call
cache
orpersist
- Shuffle. This is the block of memory used for shuffle operations (grouping, repartitioning, and
reduceByKey
. - Heap. This is where normal JVM objects are kept.
Now I would like to monitor the amount of memory in use as a % of each block by a job so I can know what I should be tuning these numbers to so that Cache and Shuffle do not spill to disk and so that Heap doesn't OOM. E.g. every few seconds I get an update like:
Cache: 40% use (40/100 GB)
Shuffle: 90% use (45/50 GB)
Heap: 10% use (1/10 GB)
I am aware I can experiment to find the sweet spots using other techniques, but I'm finding this very laboured and to just be able to monitor the usage would make writing and tuning Spark jobs much much easier.
SparkContext.getExecutorMemoryStatus
andSparkContext.getExecutorStorageStatus
. Calling them periodically would get you some of the way there...