16

I want to use a timeout on a subprocess

 from subprocess32 import check_output
 output = check_output("sleep 30", shell=True, timeout=1)

Unfortunately, whilst this raises a timeout error, it does so after 30 seconds. It seems that check_output cannot interrupt the shell command.

What can I do on on the Python side to stop this? I suspect that subprocess32 fails to kill the timed out process.

2 Answers 2

31

check_output() with timeout is essentially:

with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
    try:
        output, unused_err = process.communicate(inputdata, timeout=timeout)
    except TimeoutExpired:
        process.kill()
        output, unused_err = process.communicate()
        raise TimeoutExpired(process.args, timeout, output=output)

There are two issues:

It leads to the behaviour that you observed: the TimeoutExpired happens in a second, the shell is killed, but check_output() returns only in 30 seconds after the grandchild sleep process exits.

To workaround the issues, kill the whole process tree (all subprocesses that belong to the same group):

#!/usr/bin/env python3
import os
import signal
from subprocess import Popen, PIPE, TimeoutExpired
from time import monotonic as timer

start = timer()
with Popen('sleep 30', shell=True, stdout=PIPE, preexec_fn=os.setsid) as process:
    try:
        output = process.communicate(timeout=1)[0]
    except TimeoutExpired:
        os.killpg(process.pid, signal.SIGINT) # send signal to the process group
        output = process.communicate()[0]
print('Elapsed seconds: {:.2f}'.format(timer() - start))

Output

Elapsed seconds: 1.00
5
  • 1
    Thanks. Is this a bug in subprocess? Or is this behaviour deliberate? It's not what I expected.
    – innisfree
    Apr 30, 2016 at 13:23
  • I don't know. The current docs do not indicate that descendant processes that inherited the pipe are waited too (until EOF)—the docs mention only the child process. The behavior (timeout is not respected if descendant processes inherit the pipe) is not obvious; the docs could be updated—you could open a new issue on bugs.python.org
    – jfs
    Apr 30, 2016 at 13:40
  • not sure why, but your os.killpg() seems to kill a lot more than what I expected. If I run it through a ssh connection, the connection breaks when that method is called... Feb 9, 2017 at 9:32
  • Can you explain what the significance of preexec_fn=os.setsid is? It doesn't seem to be supported on Windows. Can this just be omitted on Windows and expect the same result? EDIT: actually, os.killpg also doesn't exist on Windows. I guess this answer is Unix-only.
    – cowlinator
    Jun 8, 2021 at 22:25
  • @cowlinator yes, the answer is Unix-only.
    – jfs
    Jun 10, 2021 at 18:33
4

Update for Python 3.6.

This is still happening but I have tested a lot of combinations of check_output, communicate and run methods and now I have a clear knowledge about where is the bug and how to avoid it in a easy way on Python 3.5 and Python 3.6.

My conclusion: It happens when you mix the use shell=True and any PIPE on stdout, stderr or stdin parameters (used in Popen and run methods).

Be careful: check_output uses PIPE inside. If you look at the code inside on Python 3.6 it is basically a call to run with stdout=PIPE: https://github.com/python/cpython/blob/ae011e00189d9083dd84c357718264e24fe77314/Lib/subprocess.py#L335

So, to solve @innisfree problem on Python 3.5 or 3.6 just do this:

check_output(['sleep', '30'], timeout=1)

And for other cases, just avoid mixing shell=True and PIPE, keeping in mind that check_output uses PIPE.

1
  • 4
    You don't need the shell to get the issue. Other processes may create their own child processes too. For example, a python script can create their own child processes using the subprocess module without starting a shell. shell=False can fix the specific example from the question but it is more than likely that the actual command is not just "sleep 30" and therefore shell=False won't fix the timeout issue.
    – jfs
    Feb 24, 2019 at 4:16

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.