9

So, I have a code which takes in input and starts a spark job in cluster.. So, something like

spark-submit driver.py -i input_path

Now, I have list of paths and I want to execute all these simulatenously..

Here is what I tried

base_command = 'spark-submit driver.py -i %s'
for path in paths:
   command = base_command%path
    subprocess.Popen(command, shell=True)

My hope was, all of the shell commands would be executed simultaneously but instead, I am noticing that it executes one command at a time..

How do i execute all the bash commands simultaneously. Thanks

8
  • Do you want to be able to tell when they are all finished?
    – Labo
    Apr 15, 2017 at 0:33
  • @Labo: naah.. just execute them all in parallel..
    – frazman
    Apr 15, 2017 at 0:34
  • What makes you think they are not executed in parallel?
    – Labo
    Apr 15, 2017 at 0:35
  • @Labo: as a test i had this base_command = 'spark-submit driver.py -i example.txt' subprocess.Popen(base_command, shell=True) print " i am here" I was hoping that it would print "i am here" but it didnt.. instead.. i was seeing all the spark log dump..
    – frazman
    Apr 15, 2017 at 0:37
  • 1
    No! My example means that subprocess.Popen launch all commands approximately at the same time and is not blocking.
    – Labo
    Apr 15, 2017 at 0:40

1 Answer 1

7

This is where pool comes in, it is designed for just this case. It maps many inputs to many threads automatically. Here is a good resource on how to use it.

from multiprocessing import Pool

def run_command(path):
    command = "spark-submit driver.py -i {}".format(path)
    subprocess.Popen(command, shell=True)

pool = Pool()
pool.map(run_command, paths)

It will create a thread for every item in paths and, run them all at the same time for the given input

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.