19

Is there any argument or options to setup a timeout for Python's subprocess.Popen method?

Something like this:

subprocess.Popen(['..'], ..., timeout=20) ?

2
  • 1
    related: subprocess with timeout
    – jfs
    May 8, 2014 at 16:29
  • Just to clarify, there is (since Python 3.3!) a timeout= parameter to the subprocess functions, so this is easy to do now. See answer @jfs linked above.
    – Ben Hoyt
    Jan 18 at 2:22

10 Answers 10

16

I would advise taking a look at the Timer class in the threading module. I used it to implement a timeout for a Popen.

First, create a callback:

def timeout( p ):
    if p.poll() is None:
        print 'Error: process taking too long to complete--terminating'
        p.kill()

Then open the process:

proc = Popen( ... )

Then create a timer that will call the callback, passing the process to it.

t = threading.Timer( 10.0, timeout, [proc] )
t.start()
t.join()

Somewhere later in the program, you may want to add the line:

t.cancel()

Otherwise, the python program will keep running until the timer has finished running.

EDIT: I was advised that there is a race condition that the subprocess p may terminate between the p.poll() and p.kill() calls. I believe the following code can fix that:

import errno

def timeout( p ):
    if p.poll() is None:
        try:
            p.kill()
            print 'Error: process taking too long to complete--terminating'
        except OSError as e:
            if e.errno != errno.ESRCH:
                raise

Though you may want to clean the exception handling to specifically handle just the particular exception that occurs when the subprocess has already terminated normally.

5
  • Mike, could you elaborate or edit a fix above? I've used similar code several times, and if there is an issue, I would definitely like to fix it. Jan 25, 2013 at 15:04
  • 1
    print 'Error: process taking too long to complete--terminating' can run even if it's a lie and your process terminates without you killing it (because it does it in the moments between your poll and kill calls). Jan 27, 2013 at 19:37
  • 2
    More importantly, if the subprocess terminates between those calls, p.kill() will raise an exception. Jan 29, 2013 at 18:50
  • 1
    PyCharm suggests its better to use if p.poll() is None than using the equality relation (==)
    – Thorben
    May 8, 2014 at 13:03
  • @Thorben, you are correct. Moreover, that is actually a PEP8 thing that PyCharm is notifying you about. In general, "Comparisons to singletons like None should always be done with is or is not, never the equality operators." legacy.python.org/dev/peps/pep-0008/… May 8, 2014 at 13:51
7

subprocess.Popen doesn't block so you can do something like this:

import time

p = subprocess.Popen(['...'])
time.sleep(20)
if p.poll() is None:
  p.kill()
  print 'timed out'
else:
  print p.communicate()

It has a drawback in that you must always wait at least 20 seconds for it to finish.

4
  • 3
    This would freeze this process for 20 seconds. Is that acceptable? Sep 17, 2010 at 7:20
  • seem it is! I think this is a little inconvenience of using subprocess.
    – sultan
    Sep 17, 2010 at 8:16
  • 1
    @aaronasterling: Popen/sleep() is a simple portable and efficient method to run an external never-ending program with a timeout. Though p.communicate() usage is problematic in this code (it might lead to loosing some of the output). See Stop reading process output in Python without hang? question for an example when Popen/sleep() is useful.
    – jfs
    Dec 11, 2013 at 16:49
  • 2
    One could reduce the drawback by using something like: wait = 0 while p.poll() is None and wait < 15: wait += 1 sleep(1) if p.poll() is None: p.kill()
    – slevon
    Mar 6, 2015 at 21:21
5
import subprocess, threading

class Command(object):
    def __init__(self, cmd):
        self.cmd = cmd
        self.process = None

    def run(self, timeout):
        def target():
            print 'Thread started'
            self.process = subprocess.Popen(self.cmd, shell=True)
            self.process.communicate()
            print 'Thread finished'

        thread = threading.Thread(target=target)
        thread.start()

        thread.join(timeout)
        if thread.is_alive():
            print 'Terminating process'
            self.process.terminate()
            thread.join()
        print self.process.returncode

command = Command("echo 'Process started'; sleep 2; echo 'Process finished'")
command.run(timeout=3)
command.run(timeout=1)

The output of this should be:

Thread started
Process started
Process finished
Thread finished
0
Thread started
Process started
Terminating process
Thread finished
-15

where it can be seen that, in the first execution, the process finished correctly (return code 0), while the in the second one the process was terminated (return code -15).

I haven't tested in windows; but, aside from updating the example command, I think it should work since I haven't found in the documentation anything that says that thread.join or process.terminate is not supported.

4

You could do

from twisted.internet import reactor, protocol, error, defer

class DyingProcessProtocol(protocol.ProcessProtocol):
    def __init__(self, timeout):
        self.timeout = timeout

    def connectionMade(self):
        @defer.inlineCallbacks
        def killIfAlive():
            try:
                yield self.transport.signalProcess('KILL')
            except error.ProcessExitedAlready:
                pass

        d = reactor.callLater(self.timeout, killIfAlive)

reactor.spawnProcess(DyingProcessProtocol(20), ...)

using Twisted's asynchronous process API.

0
3

A python subprocess auto-timeout is not built in, so you're going to have to build your own.

This works for me on Ubuntu 12.10 running python 2.7.3

Put this in a file called test.py

#!/usr/bin/python
import subprocess
import threading

class RunMyCmd(threading.Thread):
    def __init__(self, cmd, timeout):
        threading.Thread.__init__(self)
        self.cmd = cmd 
        self.timeout = timeout

    def run(self):
        self.p = subprocess.Popen(self.cmd)
        self.p.wait()

    def run_the_process(self):
        self.start()
        self.join(self.timeout)

        if self.is_alive():
            self.p.terminate()   #if your process needs a kill -9 to make 
                                 #it go away, use self.p.kill() here instead.

            self.join()

RunMyCmd(["sleep", "20"], 3).run_the_process()

Save it, and run it:

python test.py

The sleep 20 command takes 20 seconds to complete. If it doesn't terminate in 3 seconds (it won't) then the process is terminated.

el@apollo:~$  python test.py 
el@apollo:~$ 

There is three seconds between when the process is run, and it is terminated.

3

As of Python 3.3, there is also a timeout argument to the blocking helper functions in the subprocess module.

https://docs.python.org/3/library/subprocess.html

2

Unfortunately, there isn't such a solution. I managed to do this using a threaded timer that would launch along with the process that would kill it after the timeout but I did run into some stale file descriptor issues because of zombie processes or some such.

2
  • 1
    I experienced such a problem too with file descriptors on zombie processes.
    – sultan
    Sep 17, 2010 at 7:44
  • Sultan. It should be possible to correct those. I did manage to finally polish my application into something workable but it was not generic enough to publish. Sep 17, 2010 at 8:36
2

No there is no time out. I guess, what you are looking for is to kill the sub process after some time. Since you are able to signal the subprocess, you should be able to kill it too.

generic approach to sending a signal to subprocess:

proc = subprocess.Popen([command])
time.sleep(1)
print 'signaling child'
sys.stdout.flush()
os.kill(proc.pid, signal.SIGUSR1)

You could use this mechanism to terminate after a time out period.

2
  • Is it universal sys.stdout.flush() for all apps using subprocess?
    – sultan
    Sep 17, 2010 at 8:17
  • 2
    This is pretty much like Abhishek's answer. He's using SIGKILL and you're using SIGUSR1. It has the same issues. Sep 17, 2010 at 8:35
1

Yes, https://pypi.python.org/pypi/python-subprocess2 will extend the Popen module with two additional functions,

Popen.waitUpTo(timeout=seconds)

This will wait up to acertain number of seconds for the process to complete, otherwise return None

also,

Popen.waitOrTerminate

This will wait up to a point, and then call .terminate(), then .kill(), one orthe other or some combination of both, see docs for full details:

http://htmlpreview.github.io/?https://github.com/kata198/python-subprocess2/blob/master/doc/subprocess2.html

0

For Linux, you can use a signal. This is platform dependent so another solution is required for Windows. It may work with Mac though.

def launch_cmd(cmd, timeout=0):
    '''Launch an external command

    It launchs the program redirecting the program's STDIO
    to a communication pipe, and appends those responses to
    a list.  Waits for the program to exit, then returns the
    ouput lines.

    Args:
        cmd: command Line of the external program to launch
        time: time to wait for the command to complete, 0 for indefinitely
    Returns:
        A list of the response lines from the program    
    '''

    import subprocess
    import signal

    class Alarm(Exception):
        pass

    def alarm_handler(signum, frame):
        raise Alarm

    lines = []

    if not launch_cmd.init:
        launch_cmd.init = True
        signal.signal(signal.SIGALRM, alarm_handler)

    p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
    signal.alarm(timeout)  # timeout sec

    try:
        for line in p.stdout:
            lines.append(line.rstrip())
        p.wait()
        signal.alarm(0)  # disable alarm
    except:
        print "launch_cmd taking too long!"
        p.kill()

    return lines        
launch_cmd.init = False

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.