How does one use the threading and subprocess modules to spawn parallel bash processes? When I start threads ala the first answer here: How to use threading in Python?, the bash processes run sequentially instead of in parallel.
Join Stack Overflow to learn, share knowledge, and build your career.
|
|
You don't need threads to run subprocesses in parallel:
To limit number of concurrent commands you could use
This answer demonstrates various techniques to limit number of concurrent subprocesses: it shows multiprocessing.Pool, concurrent.futures, threading + Queue -based solutions. You could limit the number of concurrent child processes without using a thread/process pool:
On Unix, you could avoid the busy loop and block on |
|||||||||||||||||
|
|
A simple threading example:
Note that there are better ways to do some of this, but this is not too complicated. The example uses one thread for each command. Complexity starts to creep in when you want to do things like use a limited number of threads to handle an unknown number of commands. Those more advanced techniques don't seem too complicated once you have a grasp of threading basics. And multiprocessing gets easier once you have a handle on those techniques. |
||||
|
|
|
this is because it is supposed to do, the thing you want to do is not multithreadind but multiprocessing see this stack page |
||||
|
|