-3

I am looking for a quick answer to a very basic question related to Spark. I really don't understand how spark works and why is fast?

Question is, "Is spark fast because it divides a job into say 100 parts and run all parts at the same time or is it fast because its processing speed is superfast (in this case I am assuming that spark does not divide a job into 100 parts but just processes the job at one go) or it can do both?"

Another question, "Is spark a cluster of different physical machines or a cluster of different environments on a single machines"? Thanks,

2
  • 1
    You're question is really broad. Try searching for one of many spark introductions. In short spark can be better than R or python because it is a distributed across let's say cluster of machines that let's you process really big data fast. For small volumes of data it won't be faster than any of R, python etc. Sep 28, 2015 at 13:51
  • It is fast in its class. It means a type of tasks, type of resources (commodity hardware) and specific guarantees (like fault tolerance).
    – zero323
    Oct 5, 2015 at 23:50

1 Answer 1

0

The question is probably going to be closed, but anyway:

  • Spark may or may not partition the job, to be more precise the data, depending on configuration. It's correct that partitioning helps the with the parallelism, which provides a major performance gain. This is either non-existent or very limited in Python libraries or R.

  • A reasonably accurate explanation would be, spark is a cluster of processes which may or may not be on a single machine.

1
  • Can genetic algorithms be run faster on spark given that the search space is really big? Sep 28, 2015 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.