22

I'm working with Python 2.7 on Windows 8/XP.

I have a program A that runs another program B using the following code:

p = Popen(["B"], stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()
return

B runs a batch script C. C is a long running script and I want B to exit even though C has not finished. I have done it using the following code (in B):

p = Popen(["C"])
return

When I run B, it works as expected. When I run A however, I expected it to exit when B exits. But A waits until C exits even though B has already exitted. Any ideas on what's happening and what possible solutions could be?

Unfortunately, the obvious solution of changing A to look like B is not an option.

Here is a functional sample code to illustrate this issue: https://www.dropbox.com/s/cbplwjpmydogvu2/popen.zip?dl=1

The zip file consists of the following files with the following contents:

A.py

from subprocess import PIPE, Popen
import sys

def log(line):
    with open("log.txt", "a") as logfile:
        logfile.write(line)

log("\r\n\r\nA: I'll wait for B\r\n")

p = Popen(["C:\\Python27\\python.exe", "B.py"], stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()

log("A: Done.\r\n")
sys.exit(0)

B.py

from subprocess import Popen, PIPE
import sys

def log(line):
    with open("log.txt", "a") as logfile:
        logfile.write(line)

log("B: launching C\r\n")

p = Popen(["C.bat"])

log("B: Not waiting for C at all. bye!\r\n")
sys.exit(0)

C.bat

@echo off
echo C: Start long running task : %time% >>  "log.txt"
ping -n 10 127.0.0.1>nul
echo C: Stop long running task : %time% >>  "log.txt"

Any input is much appreciated.

6
  • If I get this right, you have program Am which runs program B which runs program C. Program A also runs program C. Is this correct? Commented Nov 6, 2012 at 5:27
  • No, program A doesn't run program C directly. It would be great if you took a look at attached example. Thanks.
    – khattam
    Commented Nov 6, 2012 at 5:32
  • So the second Popen is from program B? Commented Nov 6, 2012 at 5:35
  • Yes. A only runs B and has nothing to do with C directly.
    – khattam
    Commented Nov 6, 2012 at 15:45
  • Does supplying close_fds=True to the second Popen() call help? I'm guessing that C inherits the stdout/stderr pipes from A and thus A waits until C closes them.
    – Simon
    Commented Nov 6, 2012 at 16:48

3 Answers 3

31

You could provide start_new_session analog for the C subprocess:

#!/usr/bin/env python
import os
import sys
import platform
from subprocess import Popen, PIPE

# set system/version dependent "start_new_session" analogs
kwargs = {}
if platform.system() == 'Windows':
    # from msdn [1]
    CREATE_NEW_PROCESS_GROUP = 0x00000200  # note: could get it from subprocess
    DETACHED_PROCESS = 0x00000008          # 0x8 | 0x200 == 0x208
    kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP)  
elif sys.version_info < (3, 2):  # assume posix
    kwargs.update(preexec_fn=os.setsid)
else:  # Python 3.2+ and Unix
    kwargs.update(start_new_session=True)

p = Popen(["C"], stdin=PIPE, stdout=PIPE, stderr=PIPE, **kwargs)
assert not p.poll()

[1]: Process Creation Flags for CreateProcess()

9
  • 1
    This only works for me on windows if I had close_fds=True to the Windows kwargs line: kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP, close_fds=True) (see this answer)
    – jtpereyda
    Commented Feb 28, 2017 at 3:33
  • 1
    @jtpereyda have you noticed that it says that you cannot use close_fds=True on Windows if you redirect any of stdin,stdout, stderr?
    – jfs
    Commented Feb 28, 2017 at 8:14
  • 2
    Combining DETACHED_PROCESS and CREATE_NEW_PROCESS_GROUP makes no immediate sense. Every process is in a group in Windows, even if it's just the group for the winlogon.exe or wininit.exe session, but the only API that uses process groups is GenerateConsoleCtrlEvent for sending Ctrl+Break (or Ctrl+C if the process manually enables it, since creating a new group initially disables it) to a group of processes that are attached to one's current console. Injecting the control thread is coordinated between the console instance (conhost.exe) and the Windows session server (csrss.exe).
    – Eryk Sun
    Commented Jul 18, 2017 at 18:28
  • 1
    The equivalent of a Unix process tree in Windows is a Job object. The process should be started suspended (creation flag CREATE_SUSPENDED) to ensure it's added to the job before it can spawn other processes that leak out of the job. This works best in Windows 8+ since it supports nested jobs, whereas older versions only allow a single job per process.
    – Eryk Sun
    Commented Jul 18, 2017 at 18:36
  • 4
    Note that starting from Python 3.7, on Windows it's no longer necessary to set close_fds=True - that is now the default and works together with redirecting stdin/stdout/stderr. Details in bug tracker here: bugs.python.org/issue19764 Commented Jan 6, 2020 at 13:02
0

An alternative would be to start C as fully forked-off process in a separate process tree, e.g. via the start command of cmd.exe:

import subprocess
subprocess.Popen(["cmd.exe", "/C", "start notepad"])

Obviously, since it is completely independent, you cannot communicate with it. But you can use psutil to retrieve it's PID to at least monitor or close it, if necessary:

import psutil
for process in psutil.process_iter():
    if process.name() == 'notepad.exe':
        print(process.pid)
-1

Here is a code snippet adapted from Sebastian's answer and this answer:

#!/usr/bin/env python
import os
import sys
import platform
from subprocess import Popen, PIPE

# set system/version dependent "start_new_session" analogs
kwargs = {}
if platform.system() == 'Windows':
    # from msdn [1]
    CREATE_NEW_PROCESS_GROUP = 0x00000200  # note: could get it from subprocess
    DETACHED_PROCESS = 0x00000008          # 0x8 | 0x200 == 0x208
    kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP, close_fds=True)  
elif sys.version_info < (3, 2):  # assume posix
    kwargs.update(preexec_fn=os.setsid)
else:  # Python 3.2+ and Unix
    kwargs.update(start_new_session=True)

p = Popen(["C"], stdin=PIPE, stdout=PIPE, stderr=PIPE, **kwargs)
assert not p.poll()

I've only tested it personally on Windows.

1
  • 3
    Note that starting from Python 3.7, on Windows it's no longer necessary to set close_fds=True - that is now the default and works together with redirecting stdin/stdout/stderr. Details in bug tracker here: bugs.python.org/issue19764 Commented Jan 6, 2020 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.