6231

How do I call an external command within Python as if I had typed it in a shell or command prompt?

0

66 Answers 66

17

A simple way is to use the os module:

import os
os.system('ls')

Alternatively, you can also use the subprocess module:

import subprocess
subprocess.check_call('ls')

If you want the result to be stored in a variable try:

import subprocess
r = subprocess.check_output('ls')
16

To fetch the network id from the OpenStack Neutron:

#!/usr/bin/python
import os
netid = "nova net-list | awk '/ External / { print $2 }'"
temp = os.popen(netid).read()  /* Here temp also contains new line (\n) */
networkId = temp.rstrip()
print(networkId)

Output of nova net-list

+--------------------------------------+------------+------+
| ID                                   | Label      | CIDR |
+--------------------------------------+------------+------+
| 431c9014-5b5d-4b51-a357-66020ffbb123 | test1      | None |
| 27a74fcd-37c0-4789-9414-9531b7e3f126 | External   | None |
| 5a2712e9-70dc-4b0e-9281-17e02f4684c9 | management | None |
| 7aa697f5-0e60-4c15-b4cc-9cb659698512 | Internal   | None |
+--------------------------------------+------------+------+

Output of print(networkId)

27a74fcd-37c0-4789-9414-9531b7e3f126
1
  • You should not recommend os.popen() in 2016. The Awk script could easily be replaced with native Python code.
    – tripleee
    Commented Dec 3, 2018 at 5:49
15

Very simplest way to run any command and get the result back:

from commands import getstatusoutput

try:
    return getstatusoutput("ls -ltr")
except Exception, e:
    return None
1
14

MOST OF THE CASES:

For the most of cases, a short snippet of code like this is all you are going to need:

import subprocess
import shlex

source = "test.txt"
destination = "test_copy.txt"

base = "cp {source} {destination}'"
cmd = base.format(source=source, destination=destination)
subprocess.check_call(shlex.split(cmd))

It is clean and simple.

subprocess.check_call run command with arguments and wait for command to complete.

shlex.split split the string cmd using shell-like syntax

REST OF THE CASES:

If this do not work for some specific command, most probably you have a problem with command-line interpreters. The operating system chose the default one which is not suitable for your type of program or could not found an adequate one on the system executable path.

Example:

Using the redirection operator on a Unix system

input_1 = "input_1.txt"
input_2 = "input_2.txt"
output = "merged.txt"
base_command = "/bin/bash -c 'cat {input} >> {output}'"

base_command.format(input_1, output=output)
subprocess.check_call(shlex.split(base_command))

base_command.format(input_2, output=output)
subprocess.check_call(shlex.split(base_command))

As it is stated in The Zen of Python: Explicit is better than implicit

So if using a Python >=3.6 function, it would look something like this:

import subprocess
import shlex

def run_command(cmd_interpreter: str, command: str) -> None:
    base_command = f"{cmd_interpreter} -c '{command}'"
    subprocess.check_call(shlex.split(base_command)

12

Here are my two cents: In my view, this is the best practice when dealing with external commands...

These are the return values from the execute method...

pass, stdout, stderr = execute(["ls","-la"],"/home/user/desktop")

This is the execute method...

def execute(cmdArray,workingDir):

    stdout = ''
    stderr = ''

    try:
        try:
            process = subprocess.Popen(cmdArray,cwd=workingDir, stdout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize=1)
        except OSError:
            return [False, '', 'ERROR : command(' + ' '.join(cmdArray) + ') could not get executed!']

        for line in iter(process.stdout.readline, b''):

            try:
                echoLine = line.decode("utf-8")
            except:
                echoLine = str(line)

            stdout += echoLine

        for line in iter(process.stderr.readline, b''):

            try:
                echoLine = line.decode("utf-8")
            except:
                echoLine = str(line)

            stderr += echoLine

    except (KeyboardInterrupt,SystemExit) as err:
        return [False,'',str(err)]

    process.stdout.close()

    returnCode = process.wait()
    if returnCode != 0 or stderr != '':
        return [False, stdout, stderr]
    else:
        return [True, stdout, stderr]
2
  • 1
    Deadlock potential: use the .communicate method instead
    – pppery
    Commented Jul 7, 2016 at 2:15
  • Better yet, avoid Popen() and use the higher-level API which is now collected into the single function subprocess.run()
    – tripleee
    Commented Dec 3, 2018 at 5:27
12

Often, I use the following function for external commands, and this is especially handy for long running processes. The below method tails process output while it is running and returns the output, raises an exception if process fails.

It comes out if the process is done using the poll() method on the process.

import subprocess,sys

def exec_long_running_proc(command, args):
    cmd = "{} {}".format(command, " ".join(str(arg) if ' ' not in arg else arg.replace(' ','\ ') for arg in args))
    print(cmd)
    process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

    # Poll process for new output until finished
    while True:
        nextline = process.stdout.readline().decode('UTF-8')
        if nextline == '' and process.poll() is not None:
            break
        sys.stdout.write(nextline)
        sys.stdout.flush()

    output = process.communicate()[0]
    exitCode = process.returncode

    if (exitCode == 0):
        return output
    else:
        raise Exception(command, exitCode, output)

You can invoke it like this:

exec_long_running_proc(command = "hive", args=["-f", hql_path])
2
  • 1
    You'll get unexpected results passing an arg with space. Using repr(arg) instead of str(arg) might help by the mere coincidence that python and sh escape quotes the same way
    – sbk
    Commented May 17, 2018 at 12:08
  • 1
    @sbk repr(arg) didn't really help, the above code handles spaces as well. Now the following works exec_long_running_proc(command = "ls", args=["-l", "~/test file*"])
    – am5
    Commented Nov 17, 2018 at 0:07
11

Just to add to the discussion, if you include using a Python console, you can call external commands from IPython. While in the IPython prompt, you can call shell commands by prefixing '!'. You can also combine Python code with the shell, and assign the output of shell scripts to Python variables.

For instance:

In [9]: mylist = !ls

In [10]: mylist
Out[10]:
['file1',
 'file2',
 'file3',]
11

I wrote a small library to help with this use case:

https://pypi.org/project/citizenshell/

It can be installed using

pip install citizenshell

And then used as follows:

from citizenshell import sh
assert sh("echo Hello World") == "Hello World"

You can separate standard output from standard error and extract the exit code as follows:

result = sh(">&2 echo error && echo output && exit 13")
assert result.stdout() == ["output"]
assert result.stderr() == ["error"]
assert result.exit_code() == 13

And the cool thing is that you don't have to wait for the underlying shell to exit before starting processing the output:

for line in sh("for i in 1 2 3 4; do echo -n 'It is '; date +%H:%M:%S; sleep 1; done", wait=False)
    print ">>>", line + "!"

will print the lines as they are available thanks to the wait=False

>>> It is 14:24:52!
>>> It is 14:24:53!
>>> It is 14:24:54!
>>> It is 14:24:55!

More examples can be found at https://github.com/meuter/citizenshell

0
10

Calling an external command in Python

A simple way to call an external command is using os.system(...). And this function returns the exit value of the command. But the drawback is we won't get stdout and stderr.

ret = os.system('some_cmd.sh')
if ret != 0 :
    print 'some_cmd.sh execution returned failure'

Calling an external command in Python in background

subprocess.Popen provides more flexibility for running an external command rather than using os.system. We can start a command in the background and wait for it to finish. And after that we can get the stdout and stderr.

proc = subprocess.Popen(["./some_cmd.sh"], stdout=subprocess.PIPE)
print 'waiting for ' + str(proc.pid)
proc.wait()
print 'some_cmd.sh execution finished'
(out, err) = proc.communicate()
print 'some_cmd.sh output : ' + out

Calling a long running external command in Python in the background and stop after some time

We can even start a long running process in the background using subprocess.Popen and kill it after sometime once its task is done.

proc = subprocess.Popen(["./some_long_run_cmd.sh"], stdout=subprocess.PIPE)
# Do something else
# Now some_long_run_cmd.sh exeuction is no longer needed, so kill it
os.system('kill -15 ' + str(proc.pid))
print 'Output : ' proc.communicate()[0]
9

Use:

import subprocess

p = subprocess.Popen("df -h", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0]
print p.split("\n")

It gives nice output which is easier to work with:

['Filesystem      Size  Used Avail Use% Mounted on',
 '/dev/sda6        32G   21G   11G  67% /',
 'none            4.0K     0  4.0K   0% /sys/fs/cgroup',
 'udev            1.9G  4.0K  1.9G   1% /dev',
 'tmpfs           387M  1.4M  386M   1% /run',
 'none            5.0M     0  5.0M   0% /run/lock',
 'none            1.9G   58M  1.9G   3% /run/shm',
 'none            100M   32K  100M   1% /run/user',
 '/dev/sda5       340G  222G  100G  69% /home',
 '']
9

As an example (in Linux):

import subprocess
subprocess.run('mkdir test.dir', shell=True)

This creates test.dir in the current directory. Note that this also works:

import subprocess
subprocess.call('mkdir test.dir', shell=True)

The equivalent code using os.system is:

import os
os.system('mkdir test.dir')

Best practice would be to use subprocess instead of os, with .run favored over .call. All you need to know about subprocess is here. Also, note that all Python documentation is available for download from here. I downloaded the PDF packed as .zip. I mention this because there's a nice overview of the os module in tutorial.pdf (page 81). Besides, it's an authoritative resource for Python coders.

3
  • 2
    According to docs.python.org/2/library/…, "shell=True" may raise a security concern.
    – Nick
    Commented Mar 20, 2018 at 18:54
  • @Nick Predley: noted, but "shell=False" doesn't perform the desired function. What specifically are the security concerns and what's the alternative? Please let me know asap: I do not wish to post anything which may cause problems for anyone viewing this.
    – user8468899
    Commented Mar 21, 2018 at 19:49
  • 1
    The basic warning is in the documentation but this question explains it in more detail: stackoverflow.com/questions/3172470/…
    – tripleee
    Commented Dec 3, 2018 at 5:14
9

There are a lot of different ways to run external commands in Python, and all of them have their own plus sides and drawbacks.

My colleagues and me have been writing Python system administration tools, so we need to run a lot of external commands, and sometimes you want them to block or run asynchronously, time-out, update every second, etc.

There are also different ways of handling the return code and errors, and you might want to parse the output, and provide new input (in an expect kind of style). Or you will need to redirect standard input, standard output, and standard error to run in a different tty (e.g., when using GNU Screen).

So you will probably have to write a lot of wrappers around the external command. So here is a Python module which we have written which can handle almost anything you would want, and if not, it's very flexible so you can easily extend it:

https://github.com/hpcugent/vsc-base/blob/master/lib/vsc/utils/run.py

It doesn't work stand-alone and requires some of our other tools, and got a lot of specialised functionality over the years, so it might not be a drop-in replacement for you, but it can give you a lot of information on how the internals of Python for running commands work and ideas on how to handle certain situations.

9

For using subprocess in Python 3.5+, the following did the trick for me on Linux:

import subprocess

# subprocess.run() returns a completed process object that can be inspected
c = subprocess.run(["ls", "-ltrh"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(c.stdout.decode('utf-8'))

As mentioned in the documentation, PIPE values are byte sequences and for properly showing them decoding should be considered. For later versions of Python, text=True and encoding='utf-8' are added to kwargs of subprocess.run().

The output of the abovementioned code is:

total 113M
-rwxr-xr-x  1 farzad farzad  307 Jan 15  2018 vpnscript
-rwxrwxr-x  1 farzad farzad  204 Jan 15  2018 ex
drwxrwxr-x  4 farzad farzad 4.0K Jan 22  2018 scripts
.... # Some other lines
8

Here is calling an external command and return or print the command's output:

Python Subprocess check_output is good for

Run command with arguments and return its output as a byte string.

import subprocess
proc = subprocess.check_output('ipconfig /all')
print proc
1
  • The argument should properly be tokenized into a list, or you should explicitly pass in shell=True. In Python 3.x (where x > 3 I think) you can retrieve the output as a proper string with universal_newlines=True and you probably want to switch to subproces.run()
    – tripleee
    Commented Dec 3, 2018 at 5:22
8

If you need to call a shell command from a Python notebook (like Jupyter, Zeppelin, Databricks, or Google Cloud Datalab) you can just use the ! prefix.

For example,

!ls -ilF
8

Use subprocess.call:

from subprocess import call

# Using list
call(["echo", "Hello", "world"])

# Single string argument varies across platforms so better split it
call("echo Hello world".split(" "))
8

If you're writing a Python shell script and have IPython installed on your system, you can use the bang prefix to run a shell command inside IPython:

!ls
filelist = !ls
1
  • @PeterMortensen I don't think it works in DOS, but it should work in Cygwin. Commented Nov 30, 2019 at 1:39
7

After some research, I have the following code which works very well for me. It basically prints both standard output and standard error in real time.

stdout_result = 1
stderr_result = 1


def stdout_thread(pipe):
    global stdout_result
    while True:
        out = pipe.stdout.read(1)
        stdout_result = pipe.poll()
        if out == '' and stdout_result is not None:
            break

        if out != '':
            sys.stdout.write(out)
            sys.stdout.flush()


def stderr_thread(pipe):
    global stderr_result
    while True:
        err = pipe.stderr.read(1)
        stderr_result = pipe.poll()
        if err == '' and stderr_result is not None:
            break

        if err != '':
            sys.stdout.write(err)
            sys.stdout.flush()


def exec_command(command, cwd=None):
    if cwd is not None:
        print '[' + ' '.join(command) + '] in ' + cwd
    else:
        print '[' + ' '.join(command) + ']'

    p = subprocess.Popen(
        command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd
    )

    out_thread = threading.Thread(name='stdout_thread', target=stdout_thread, args=(p,))
    err_thread = threading.Thread(name='stderr_thread', target=stderr_thread, args=(p,))

    err_thread.start()
    out_thread.start()

    out_thread.join()
    err_thread.join()

    return stdout_result + stderr_result
1
  • 4
    your code may lose data when the subprocess exits while there is some data is buffered. Read until EOF instead, see teed_call()
    – jfs
    Commented Jul 13, 2015 at 18:52
7

Update 2015: Python 3.5 added subprocess.run which is much easier to use than subprocess.Popen. I recommend that.

>>> subprocess.run(["ls", "-l"])  # doesn't capture output
CompletedProcess(args=['ls', '-l'], returncode=0)

>>> subprocess.run("exit 1", shell=True, check=True)
Traceback (most recent call last):
  ...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1

>>> subprocess.run(["ls", "-l", "/dev/null"], capture_output=True)
CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0,
stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n', stderr=b'')
4
  • 10
    Deprecated doesn't only mean "isn't developed anymore" but also "you are discouraged from using this". Deprecated features may break anytime, may be removed anytime, or may dangerous. You should never use this in important code. Deprecation is merely a better way than removing a feature immediately, because it gives programmers the time to adapt and replace their deprecated functions.
    – Misch
    Commented Apr 19, 2013 at 8:07
  • 4
    Just to prove my point: "Deprecated since version 2.6: The commands module has been removed in Python 3. Use the subprocess module instead."
    – Misch
    Commented Apr 19, 2013 at 8:14
  • It's not dangerous! The Python devs are careful only to break features between major releases (ie. between 2.x and 3.x). I've been using the commands module since 2004's Python 2.4. It works the same today in Python 2.7. Commented Apr 23, 2013 at 16:09
  • 6
    With dangerous, I didn't mean that it may be removed anytime (that's a different problem), neither did I say that it is dangerous to use this specific module. However it may become dangerous if a security vulnerability is discovered but the module isn't further developed or maintained. (I don't want to say that this module is or isn't vulnerable to security issues, just talking about deprecated stuff in general)
    – Misch
    Commented Apr 23, 2013 at 16:23
6

For Python 3.5+ it is recommended that you use the run function from the subprocess module. This returns a CompletedProcess object, from which you can easily obtain the output as well as return code.

from subprocess import PIPE, run

command = ['echo', 'hello']
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True)
print(result.returncode, result.stdout, result.stderr)
1
  • 3
    answer with run function was added in 2015 year. You repeated it. I think it was a reason of down vote Commented Mar 11, 2017 at 18:27
5

If you are not using user input in the commands, you can use this:

from os import getcwd
from subprocess import check_output
from shlex import quote

def sh(command):
    return check_output(quote(command), shell=True, cwd=getcwd(), universal_newlines=True).strip()

And use it as

branch = sh('git rev-parse --abbrev-ref HEAD')

shell=True will spawn a shell, so you can use pipe and such shell things sh('ps aux | grep python'). This is very very handy for running hardcoded commands and processing its output. The universal_lines=True make sure the output is returned in a string instead of binary.

cwd=getcwd() will make sure that the command is run with the same working directory as the interpreter. This is handy for Git commands to work like the Git branch name example above.

Some recipes

  • free memory in megabytes: sh('free -m').split('\n')[1].split()[1]
  • free space on / in percent sh('df -m /').split('\n')[1].split()[4][0:-1]
  • CPU load sum(map(float, sh('ps -ef -o pcpu').split('\n')[1:])

But this isn't safe for user input, from the documentation:

Security Considerations

Unlike some other popen functions, this implementation will never implicitly call a system shell. This means that all characters, including shell metacharacters, can safely be passed to child processes. If the shell is invoked explicitly, via shell=True, it is the application’s responsibility to ensure that all whitespace and metacharacters are quoted appropriately to avoid shell injection vulnerabilities.

When using shell=True, the shlex.quote() function can be used to properly escape whitespace and shell metacharacters in strings that are going to be used to construct shell commands.

Even using the shlex.quote(), it is good to keep a little paranoid when using user inputs on shell commands. One option is using a hardcoded command to take some generic output and filtering by user input. Anyway using shell=False will make sure that only the exactly process that you want to execute will be executed or you get a No such file or directory error.

Also there is some performance impact on shell=True, from my tests it seems about 20% slower than shell=False (the default).

In [50]: timeit("check_output('ls -l'.split(), universal_newlines=True)", number=1000, globals=globals())
Out[50]: 2.6801227919995654

In [51]: timeit("check_output('ls -l', universal_newlines=True, shell=True)", number=1000, globals=globals())
Out[51]: 3.243950183999914
5

You can try using os.system() for running external commands.

Example:

import os

try:
  os.system('ls')
  pass
except:
  print("Error running command")
  pass

In the example, the script imports os and tries to run the command listed in os.system(). If the command was to fail then it would print "Error running command" without the script stopping due to the error.

And yes, it’s just that simple!

4

Using the Popen function of the subprocess Python module is the simplest way of running Linux commands. In that, the Popen.communicate() function will give your commands output. For example

import subprocess

..
process = subprocess.Popen(..)   # Pass command and arguments to the function
stdout, stderr = process.communicate()   # Get command output and error
..
1
  • This is no longer true, and probably wasn't when this answer was posted. You should prefer subprocess.check_call() and friends unless you absolutely need the lower-level control of the more-complex Popen(). In recent Python versions, the go-to workhorse is subprocess.run()
    – tripleee
    Commented Dec 3, 2018 at 5:30
4

There are many ways to call a command.

  • For example:

if and.exe needs two parameters. In cmd we can call sample.exe use this: and.exe 2 3 and it show 5 on screen.

If we use a Python script to call and.exe, we should do like..

  1. os.system(cmd,...)

    • os.system(("and.exe" + " " + "2" + " " + "3"))
  2. os.popen(cmd,...)

    • os.popen(("and.exe" + " " + "2" + " " + "3"))
  3. subprocess.Popen(cmd,...)
    • subprocess.Popen(("and.exe" + " " + "2" + " " + "3"))

It's too hard, so we can join cmd with a space:

import os
cmd = " ".join(exename,parameters)
os.popen(cmd)
1
  • os.popen should not be recommended and perhaps even mentioned any longer. The subpocess example should pass the arguments as a list instead of joining them with spaces.
    – tripleee
    Commented Dec 3, 2018 at 5:25
4
import subprocess

p = subprocess.run(["ls", "-ltr"], capture_output=True)
print(p.stdout.decode(), p.stderr.decode())

Try online

2
  • An explanation would be in order. E.g, what is the idea and how is it different from the previous 50 answers 11 years later? Commented Apr 7, 2021 at 17:49
  • This is identical to your own answer from a few days earlier only with even less detail.
    – tripleee
    Commented Jun 9, 2021 at 19:34
4

os.popen() is the easiest and the most safest way to execute a command. You can execute any command that you run on the command line. In addition you will also be able to capture the output of the command using os.popen().read()

You can do it like this:

import os
output = os.popen('Your Command Here').read()
print (output)

An example where you list all the files in the current directory:

import os
output = os.popen('ls').read()
print (output)
# Outputs list of files in the directory
3

I would recommend the following method 'run' and it will help us in getting standard output, standard error and exit status as a dictionary; the caller of this can read the dictionary return by 'run' method to know the actual state of the process.

  def run (cmd):
       print "+ DEBUG exec({0})".format(cmd)
       p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True, shell=True)
       (out, err) = p.communicate()
       ret        = p.wait()
       out        = filter(None, out.split('\n'))
       err        = filter(None, err.split('\n'))
       ret        = True if ret == 0 else False
       return dict({'output': out, 'error': err, 'status': ret})
  #end
2
  • 1
    This incompletely reimplements something like subprocess.run(). You should particularly avoid shell=True when it's not strictly necessary.
    – tripleee
    Commented Dec 3, 2018 at 5:51
  • What is #end for? Commented Apr 7, 2021 at 17:34
3

There are a number of ways of calling an external command from Python. There are some functions and modules with the good helper functions that can make it really easy. But the recommended thing among all is the subprocess module.

import subprocess as s
s.call(["command.exe", "..."])

The call function will start the external process, pass some command line arguments and wait for it to finish. When it finishes you continue executing. Arguments in call function are passed through the list. The first argument in the list is the command typically in the form of an executable file and subsequent arguments in the list whatever you want to pass.

If you have called processes from the command line in the windows before, you'll be aware that you often need to quote arguments. You need to put quotations mark around it. If there's a space then there's a backslash and there are some complicated rules, but you can avoid a whole lot of that in Python by using subprocess module because it is a list and each item is known to be a distinct and python can get quoting correctly for you.

In the end, after the list, there are a number of optional parameters one of these is a shell and if you set shell equals to true then your command is going to be run as if you have typed in at the command prompt.

s.call(["command.exe", "..."], shell=True)

This gives you access to functionality like piping, you can redirect to files, you can call multiple commands in one thing.

One more thing, if your script relies on the process succeeding then you want to check the result and the result can be checked with the check call helper function.

s.check_call(...)

It is exactly the same as a call function, it takes the same arguments, takes the same list, you can pass in any of the extra arguments but it going to wait for the functions to complete. And if the exit code of the function is anything other then zero, it will through an exception in the python script.

Finally, if you want tighter control Popen constructor which is also from the subprocess module. It also takes the same arguments as incall & check_call function but it returns an object representing the running process.

p=s.Popen("...")

It does not wait for the running process to finish also it's not going to throw any exception immediately but it gives you an object that will let you do things like wait for it to finish, let you communicate to it, you can redirect standard input, standard output if you want to display output somewhere else and a lot more.

3

You can run any command using Popen from the subprocess module.

from subprocess import Popen

First of all, a command object is created with all arguments which you want to run. For example, in the snippet below, the gunicorm command object has been formed with all the arguments:

cmd = (
        "gunicorn "
        "-c gunicorn_conf.py "
        "-w {workers} "
        "--timeout {timeout} "
        "-b {address}:{port} "
        "--limit-request-line 0 "
        "--limit-request-field_size 0 "
        "--log-level debug "
        "--max-requests {max_requests} "
        "manage:app").format(**locals())

Then this command object is used with Popen to instantiate a process:

process = Popen(cmd, shell=True)

This process can be terminated as well based upon any signal, using the code line below:

Popen.terminate(process)

And you can wait till the completion of above command's execution:

process.wait()
3

Here there are a lot of answers, but none fulfilled all my needs.

  • I need to run the command and capture the output and exit code.
  • I need to timeout the executed program and force it to exit if timeout is reached, and kill all its child processes.
  • and I need that it works in Windows XP and later, Cygwin and Linux. In Python 2 and 3.

So I created this:

def _run(command, timeout_s=False, shell=False):
    ### run a process, capture the output and wait for it to finish. if timeout is specified then Kill the subprocess and its children when the timeout is reached (if parent did not detach)
    ## usage: _run(arg1, arg2, arg3)
        # arg1: command + arguments. Always pass a string; the function will split it when needed
        # arg2: (optional) timeout in seconds before force killing
        # arg3: (optional) shell usage. default shell=False
    ## return: a list containing: exit code, output, and if timeout was reached or not

    # - Tested on Python 2 and 3 on Windows XP, Windows 7, Cygwin and Linux.
    # - preexec_fn=os.setsid (py2) is equivalent to start_new_session (py3) (works on Linux only), in Windows and Cygwin we use TASKKILL
    # - we use stderr=subprocess.STDOUT to merge standard error and standard output
    import sys, subprocess, os, signal, shlex, time

    def _runPY3(command, timeout_s=None, shell=False):
        # py3.3+ because: timeout was added to communicate() in py3.3.
        new_session=False
        if sys.platform.startswith('linux'): new_session=True
        p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, start_new_session=new_session, shell=shell)

        try:
            out = p.communicate(timeout=timeout_s)[0].decode('utf-8')
            is_timeout_reached = False
        except subprocess.TimeoutExpired:
            print('Timeout reached: Killing the whole process group...')
            killAll(p.pid)
            out = p.communicate()[0].decode('utf-8')
            is_timeout_reached = True
        return p.returncode, out, is_timeout_reached

    def _runPY2(command, timeout_s=0, shell=False):
        preexec=None
        if sys.platform.startswith('linux'): preexec=os.setsid
        p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, preexec_fn=preexec, shell=shell)

        start_time = time.time()
        is_timeout_reached = False
        while timeout_s and p.poll() == None:
            if time.time()-start_time >= timeout_s:
                print('Timeout reached: Killing the whole process group...')
                killAll(p.pid)
                is_timeout_reached = True
                break
            time.sleep(1)
        out = p.communicate()[0].decode('utf-8')
        return p.returncode, out, is_timeout_reached

    def killAll(ParentPid):
        if sys.platform.startswith('linux'):
            os.killpg(os.getpgid(ParentPid), signal.SIGTERM)
        elif sys.platform.startswith('cygwin'):
            # subprocess.Popen(shlex.split('bash -c "TASKKILL /F /PID $(</proc/{pid}/winpid) /T"'.format(pid=ParentPid)))
            winpid=int(open("/proc/{pid}/winpid".format(pid=ParentPid)).read())
            subprocess.Popen(['TASKKILL', '/F', '/PID', str(winpid), '/T'])
        elif sys.platform.startswith('win32'):
            subprocess.Popen(['TASKKILL', '/F', '/PID', str(ParentPid), '/T'])

    # - In Windows, we never need to split the command, but in Cygwin and Linux we need to split if shell=False (default), shlex will split the command for us
    if shell==False and (sys.platform.startswith('cygwin') or sys.platform.startswith('linux')):
        command=shlex.split(command)

    if sys.version_info >= (3, 3): # py3.3+
        if timeout_s==False:
            returnCode, output, is_timeout_reached = _runPY3(command, timeout_s=None, shell=shell)
        else:
            returnCode, output, is_timeout_reached = _runPY3(command, timeout_s=timeout_s, shell=shell)
    else:  # Python 2 and up to 3.2
        if timeout_s==False:
            returnCode, output, is_timeout_reached = _runPY2(command, timeout_s=0, shell=shell)
        else:
            returnCode, output, is_timeout_reached = _runPY2(command, timeout_s=timeout_s, shell=shell)

    return returnCode, output, is_timeout_reached

Then use it like this:

Always pass the command as one string (it is easier). You do not need to split it; the function will split it when needed.

If your command works in your shell, it will work with this function, so test your command in your shell first cmd/Bash.

So we can use it like this with a timeout:

a=_run('cmd /c echo 11111 & echo 22222 & calc',3)
for i in a[1].splitlines(): print(i)

Or without a timeout:

b=_run('cmd /c echo 11111 & echo 22222 & calc')

More examples:

b=_run('''wmic nic where 'NetConnectionID="Local Area Connection"' get NetConnectionStatus /value''')
print(b)

c=_run('cmd /C netsh interface ip show address "Local Area Connection"')
print(c)

d=_run('printf "<%s>\n" "{foo}"')
print(d)

You can also specify shell=True, but it is useless in most cases with this function. I prefer to choose myself the shell I want, but here it is if you need it too:

# windows
e=_run('echo 11111 & echo 22222 & calc',3, shell=True)
print(e)
# Cygwin/Linux:
f=_run('printf "<%s>\n" "{foo}"', shell=True)
print(f)

Why did I not use the simpler new method subprocess.run()?

  • because it is supported in Python 3.7+, but the last supported Python version in Windows XP is 3.4.
  • and because the timeout argument of this function is useless in Windows, it does not kill the child processes of the executed command.
  • if you use the capture_output + timeout argument, it will hang if there is a child process still running. And it is still broken in Windows, for which the issue 31447 is still open.
1
  • 1
    I removed the "2022 answer" because it’s misleading. In 2022 you don’t support winxp nor Python 2 and you use subprocess.run(). The question is not "how do I run system commands on old platforms" but "how do I run system commands", period.
    – bfontaine
    Commented May 24, 2022 at 10:08

Not the answer you're looking for? Browse other questions tagged or ask your own question.