18

I am writing some multiprocessing code (Python 2.6.4, WinXP) that spawns processes to run background tasks. In playing around with some trivial examples, I am running into an issue where my code just continuously spawns new processes, even though I only tell it to spawn a fixed number.

The program itself runs fine, but if I look in Windows TaskManager, I keep seeing new 'python.exe' processes appear. They just keep spawning more and more as the program runs (eventually starving my machine).


For example,
I would expect the code below to launch 2 python.exe processes. The first being the program itself, and the second being the child process it spawns. Any idea what I am doing wrong?

import time
import multiprocessing


class Agent(multiprocessing.Process):
    def __init__(self, i):
        multiprocessing.Process.__init__(self)
        self.i = i

    def run(self):
        while True:
            print 'hello from %i' % self.i
            time.sleep(1)


agent = Agent(1)
agent.start()

3 Answers 3

24

It looks like you didn't carefully follow the guidelines in the documentation, specifically this section where it talks about "Safe importing of main module".

You need to protect your launch code with an if __name__ == '__main__': block or you'll get what you're getting, I believe.

I believe it comes down to the multiprocessing module not being able to use os.fork() as it does on Linux, where an already-running process is basically cloned in memory. On Windows (which has no such fork()) it must run a new Python interpreter and tell it to import your main module and then execute the start/run method once that's done. If you have code at "module level", unprotected by the name check, then during the import it starts the whole sequence over again, ad infinitum

3
  • 1
    it's not weird once you understand the implementation, which is exactly what Peter points out.
    – jnoller
    Commented Dec 18, 2009 at 0:57
  • @jnoller, it's not obvious unless you look carefully, but corey made his comment before I edited my answer to add the clarification. Commented Dec 18, 2009 at 1:02
  • 3
    From Python 3.3.0 the behavior changed and now instead of recursive starting of new processes one gets RuntimeError with message Attempt to start a new process before the current process has finished its bootstrapping phase. See bugs.python.org/issue15646 Commented Sep 24, 2012 at 18:49
0

When I run this in Linux with python2.6, I see a maximum of 4 python2.6 processes and I can't guarantee that they're all from this process. They're definitely not filling up the machine.

Need new python version? Linux/Windows difference?

2
  • If you want to see the relationships between processes in general on linux you can use "ps axjf" to get some sort of "tree" view of processes. Commented Dec 17, 2009 at 18:58
  • i'm on the newest Python 2.x (2.6.4) Commented Dec 17, 2009 at 19:04
0

I don't see anything wrong with that. Works fine on Ubuntu 9.10 (Python 2.6.4).

Are you sure you don't have cron or something starting multiple copies of your script? Or that the spawned script is not calling anything that would start a new instance, for example as a side effect of import if your code runs directly on import?

1
  • i'm positive only a single version is starting. this is totally bizarre. It just keeps spawning and spawning on my machine. thanks for verifying on *nix. Commented Dec 17, 2009 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.