0

I read a post about Storm below. What is the "task" in Storm parallelism

Both the question and the answer are very helpful for me, a beginner of Storm. However I have still some comments. It is said in the answer,

Running more than one task per executor does not increase the level of parallelism -- an >executor always has one thread that it uses for all of its tasks, which means that tasks >run serially on an executor.

Does it mean the "parallel computing" of Storm concentrate on "parallel machines". As Miguno mentioned, it gives us the flexibility to expand cluster machines. However the machine is becoming more and more CPU cores integrated, could it be more effeient if we use parallel computing against each machine in the cluster? (To some extent I felt the multi cores in each machine is also a "cluster"). There are also a couple of very helpful parallel programming libraries like AKKA, Disruptor, Killim and etc, which make us develop parallel programms conviniently.

I'm not sure if I misunderstand something about the Storm frame (or this kind of Could computing frame)?

Thanks,

1 Answer 1

0

Bit confused what exactly you are asking. As mentioned in the same article that by increasing the Worker Process you can increase the number of Java process to run across the cluster for executing the tasks.

From the wiki it says

Config.TOPOLOGY_WORKERS: This sets the number of worker processes to use to execute the topology. For example, if you set this to 25, there will be 25 Java processes across the cluster executing all the tasks. If you had a combined 150 parallelism across all components in the topology, each worker process will have 6 tasks running within it as threads.

It can be done by doing something like

    Config conf = new Config();
    conf.setNumWorkers(25); 

By saying parallel computing if you intent to increase the number of Threads then you should be looking at the executors, by increasing the same you can spawn more threads per component. From the same article you mentioned it says

Note that as of Storm 0.8 the parallelism_hint parameter now specifies the initial number of executors (not tasks!) for that bolt.

e.g topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2)

please correct me if I misunderstood what you are seeking

1
  • Thanks for the reply. Yes this answer is what I was seeking (Sorry I missed the original artical you mentioned). Another confusing for "Cloud Computing" is I felt multi-thread programming is not easy, because there are so many posts under "multithreading" Tag here:). However in the articles of "Cloud Computing" I have seen, there pay less attention on this, only on the ability of expanding architecture. Maybe I lost some articles. (I did not have practise experience of Could Computing.)
    – Patrick Yu
    Oct 26, 2013 at 6:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.