9

I'm running into some difficulties getting output from a subprocess stdout pipe. I'm launching some third party code via it, in order to extract log output. Up until a recent update of the third party code, everything worked fine. After the update, python has started blocking indefinitely, and not actually showing any output. I can manually launch the third party app fine and see output.

A basic version of the code I'm using:

import subprocess, time
from threading import Thread

def enqueue_output(out):
    print "Hello from enqueue_output"
    for line in iter(out.readline,''):
        line = line.rstrip("\r\n")
        print "Got %s" % line
    out.close()

proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=1)
thread = Thread(target=enqueue_output, args=(proc.stdout,))
thread.daemon = True
thread.start()

time.sleep(30)

This works perfectly if I substitute third_party.exe for this script:

import time, sys

while True:
    print "Test"
    sys.stdout.flush()
    time.sleep(1)

So I'm unclear as to magic needs to be done to get this working with the original command.

These are all variants of the subprocess.Popen line I've tried with no success:

proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=0)
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, shell=True)
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, creationflags=subprocess.CREATE_NEW_CONSOLE)
si = subprocess.STARTUPINFO()
si.dwFlags = subprocess.STARTF_USESTDHANDLES | subprocess.STARTF_USESHOWWINDOW
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, startupinfo=si)

Edit 1: I can't actually use .communicate() in this case. The app I'm launching remains running for long periods of time (days to weeks). The only way I could actually test .communicate() would be to kill the app shortly after it launches, which I don't feel would give me valid results.

Even the non-threaded version of this fails:

import subprocess, time
from threading import Thread

proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, stderr=subprocess.PIPE)

print "App started, reading output..."
for line in iter(proc.stdout.readline,''):
    line = line.rstrip("\r\n")
    print "Got: %s" % line

Edit 2: Thanks to jdi, the following works okay:

import tempfile, time, subprocess

w = "test.txt"
f = open("test.txt","a")
p = subprocess.Popen("third_party.exe", shell=True, stdout=f,
                        stderr=subprocess.STDOUT, bufsize=0)

time.sleep(30)

with open("test.txt", 'r') as r:
    for line in r:
        print line
f.close()
5
  • Is it possible that your third party program switched to writing to stderr?
    – jdi
    Commented May 2, 2012 at 1:33
  • Doesn't seem to be the case. If I intercept stderr instead, the same thing happens (though, I do see the app's output on the console, but it's just printing, it's not going through Python).
    – devicenull
    Commented May 2, 2012 at 1:36
  • It sounds as if the app has stopped using standard output and is writing directly to the console. If so, there's not very much you can do about this. Are you able to check with the app's vendor? Commented May 2, 2012 at 2:05
  • One last-resort possibility is to read the console content directly, see my answer to stackoverflow.com/questions/7990851/… Commented May 2, 2012 at 2:05
  • The app's vendor has informed me they switched from 'std::wcout' to c-style stdout. I'm unclear on exactly what that means.
    – devicenull
    Commented May 2, 2012 at 2:21

2 Answers 2

6

First I would recommend that you simplify this example to make sure you can actually read anything. Remove the complication of the thread from the mix:

proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=1)
print proc.communicate()

If that works, great. Then you are having problems possibly with how you are reading the stdout directly or possibly in your thread.

If this does not work, have you tried piping stderr to stdout as well?

proc = subprocess.Popen("third_party.exe", 
                        stdout=subprocess.PIPE, 
                        stderr=subprocess.STDOUT, bufsize=1)

Update

Since you say communicate() is deadlocking, here is another approach you can try to see if its a problem with the internal buffer of subprocess...

import tempfile
import subprocess

w = tempfile.NamedTemporaryFile()
p = subprocess.Popen('third_party.exe', shell=True, stdout=w, 
                        stderr=subprocess.STDOUT, bufsize=0)

with open(w.name, 'r') as r:
    for line in r:
        print line
w.close()
5
  • If I had an error with the thread, why would it work for my simple test app, but not the real thing? I can't use communicate, as the app doesn't actually end. It's a server, and stays running for long periods of time.
    – devicenull
    Commented May 2, 2012 at 1:38
  • @devicenull: I would say the difference is that your test script is printing 4 characters, slowly, every second, while I have no idea what kind of output your 3rd party app is doing. You could be deadlocking from the buffer. Also, when you have problems like this, its always best to remove as many variables as possible and make sure each step works correctly.
    – jdi
    Commented May 2, 2012 at 1:40
  • Outputting to a file works perfectly. Any ideas why that works, but pipes don't?
    – devicenull
    Commented May 2, 2012 at 2:20
  • @devicenull: This is technically the same thing, only using a named file. I cant remember where I have seen it before, but I have read about the internal buffer of subprocess hitting limits with certain amounts of large output. The workaround was to use your own file descriptors.
    – jdi
    Commented May 2, 2012 at 2:22
  • @devicenull: the key difference is that your edit1 uses stderr=PIPE while edit2 uses stderr=STDOUT. And (perhaps) shell=True might make a difference -- try with and without it. If you use stderr=STDOUT then your edit1 might work -- you don't need to use temporary files here.
    – jfs
    Commented Oct 13, 2015 at 12:49
2
args = ['svn','log','-v']

def foo(info=''):
    import logging
    import subprocess
    import tempfile
    try:
        pipe = subprocess.Popen(args,bufsize = 0,\
            stdout = subprocess.PIPE,\
            stderr=subprocess.STDOUT)
    except Exception as e:
        logging.error(str(e))
        return False
    while 1:
        s = pipe.stdout.read()
        if s:
            print s,
        if pipe.returncode is None:
            pipe.poll()
        else:
            break
    if not 0 == pipe.returncode:
        return False
    return True

print foo()

This one should works,not thread,temp file magic.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.