2

I read the following on the Dask documentation in the known limitations section:

  • It [Dask] is not fault tolerant. The failure of any worker is likely to crash the system.

  • It does not fail gracefully in case of errors

but I don't see any mentions of fault tolerance in the comparison with Spark. These are currently the "Reasons why you might choose Spark":

  • You prefer Scala or the SQL language
  • You have mostly JVM infrastructure and legacy systems
  • You want an established and trusted solution for business
  • You are mostly doing business analytics with some lightweight machine learning
  • You want an all-in-one solution

My questions:

  • Is Spark actually designed for fault tolerance in a manner that Dask currently isn't?
  • What type of fault tolerance does Spark provide (in theory/practice) that Dask doesn't, if any, or viceversa?

2 Answers 2

6

That set of doc pages was very very old and should not have been publicly available. I have just removed them. Please see http://dask.pydata.org/en/latest/ for up-to-date documentation.

Dask is fault tolerant to the loss of any worker. It will fail if the central scheduler fails.

0
1

I'm currently loading around 36 million of records to DynamoDB using dask bags and I'm struggling with the fact the dask processes get frozen. I splited them into 2240 independent process in order to track each process and I can say that sometimes those small process get frozen too. I'd guess then that Dask is not Fault Tolerance at least not good enough based on my experience.

dask==2.30.0

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.