51

I'm trying to learn twitter storm by following the great article "Understanding the parallelism of a Storm topology"

However I'm a bit confused by the concept of "task". Is a task an running instance of the component(spout or bolt) ? A executor having multiple tasks actually is saying the same component is executed for multiple times by the executor, am I correct ?

Moreover in a general parallelism sense, Storm will spawn a dedicated thread(executor) for a spout or bolt, but what is contributed to the parallelism by an executor(thread) having multiple tasks ? I think having multiple tasks in a thread, since a thread executes sequentially, only make the thread a kind of "cached" resource, which avoids spawning new thread for next task run. Am I correct?

I may clear those confusion by myself after taking more time to investigate, but you know, we both love stackoverflow ;-)

Thanks in advance.

1
  • 1
    I've read the documentation thrice to clear the same confusion and you solved my problem.
    – pavan
    Nov 13, 2014 at 14:30

1 Answer 1

78

Disclaimer: I wrote the article you referenced in your question above.

However I'm a bit confused by the concept of "task". Is a task an running instance of the component(spout or bolt) ? A executor having multiple tasks actually is saying the same component is executed for multiple times by the executor, am I correct ?

Yes, and yes.

Moreover in a general parallelism sense, Storm will spawn a dedicated thread(executor) for a spout or bolt, but what is contributed to the parallelism by an executor(thread) having multiple tasks ?

Running more than one task per executor does not increase the level of parallelism -- an executor always has one thread that it uses for all of its tasks, which means that tasks run serially on an executor.

As I wrote in the article please note that:

  • The number of executor threads can be changed after the topology has been started (see storm rebalance command).
  • The number of tasks of a topology is static.

And by definition there is the invariant of #executors <= #tasks.

So one reason for having 2+ tasks per executor thread is to give you the flexibility to expand/scale up the topology through the storm rebalance command in the future without taking the topology offline. For instance, imagine you start out with a Storm cluster of 15 machines but already know that next week another 10 boxes will be added. Here you could opt for running the topology at the anticipated parallelism level of 25 machines already on the 15 initial boxes (which is of course slower than 25 boxes). Once the additional 10 boxes are integrated you can then storm rebalance the topology to make full use of all 25 boxes without any downtime.

Another reason to run 2+ tasks per executor is for (primarily functional) testing. For instance, if your dev machine or CI server is only powerful enough to run, say, 2 executors alongside all the other stuff running on the machine, you can still run 30 tasks (here: 15 per executor) to see whether code such as your custom Storm grouping is working as expected.

In practice we normally we run 1 task per executor.

PS: Note that Storm will actually spawn a few more threads behind the scenes. For instance, each executor has its own "send thread" that is responsible for handling outgoing tuples. There are also "system-level" background threads for e.g. acking tuples that run alongside "your" threads. IIRC the Storm UI counts those acking threads in addition to "your" threads.

3
  • @miguno I have a question, let's say that you have initially a bolt that performs a task (like a group-by operation on a number batched tuples) how can I scale this up vertically? From my understanding each worker is essentially a "mirror" of the topology but how can I scale it up vertically to distribute the load across all cluster boxes. In this case I would want to change the number of bolts performing the group-by operation from 1 to 2 (or more). Your input to clarify this would be awesome.
    – jtimz
    Jul 17, 2014 at 17:26
  • Storm's scaling model uses horizontal scaling, similar to other processing technologies like Kafka Streams, ksqlDB, or Spark. That is, you use more workers for scale-out, and use less workers for scale-in. This is how you distributed the load across all cluster boxes. (In comparison, vertical scaling is given more resources to the same number of workers, like giving them faster CPUs or more RAM.)
    – miguno
    Aug 12, 2020 at 10:08
  • To go from 1 bolt to 2 bolts, for example, you have to stop your Storm topology, re-configure its parallelism settings ("Now use 2 bolts!"), and then restart/resubmit your topology. Unlike say, Kafka Streams, Storm cannot elastically scale-in or scale-out live during runtime -- it always requires taking down your topology for re-configuration.
    – miguno
    Aug 12, 2020 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.