In spark 2.x if the driver dies does the executors die too? If yes, if restarted will the whole job start again wid reference from check point directory?
1 Answer
Driver is a single point of failure in Spark applications. If driver dies, all other linked components will be discarded as well.
In applications using checkpointing for reliability (streaming), checkpoints can be use to recover state. Please refer to:
- Spark Streaming Programming Guide - Checkpointing
- Structured Streaming Programming Guide - Recovering from Failures with Checkpointing
for details.
-
Thanks that helped. I am still confused how fault tolerance and exactly once strategy is ensured when I am using window operation on streaming job and one of the executors fail. Jan 29, 2018 at 22:39