I read the following on the Dask documentation in the known limitations section:
It [Dask] is not fault tolerant. The failure of any worker is likely to crash the system.
It does not fail gracefully in case of errors
but I don't see any mentions of fault tolerance in the comparison with Spark. These are currently the "Reasons why you might choose Spark":
- You prefer Scala or the SQL language
- You have mostly JVM infrastructure and legacy systems
- You want an established and trusted solution for business
- You are mostly doing business analytics with some lightweight machine learning
- You want an all-in-one solution
My questions:
- Is Spark actually designed for fault tolerance in a manner that Dask currently isn't?
- What type of fault tolerance does Spark provide (in theory/practice) that Dask doesn't, if any, or viceversa?