How do I call an external command within Python as if I had typed it in a shell or command prompt?
66 Answers
A simple way is to use the os module:
import os
os.system('ls')
Alternatively, you can also use the subprocess module:
import subprocess
subprocess.check_call('ls')
If you want the result to be stored in a variable try:
import subprocess
r = subprocess.check_output('ls')
To fetch the network id from the OpenStack Neutron:
#!/usr/bin/python
import os
netid = "nova net-list | awk '/ External / { print $2 }'"
temp = os.popen(netid).read() /* Here temp also contains new line (\n) */
networkId = temp.rstrip()
print(networkId)
Output of nova net-list
+--------------------------------------+------------+------+
| ID | Label | CIDR |
+--------------------------------------+------------+------+
| 431c9014-5b5d-4b51-a357-66020ffbb123 | test1 | None |
| 27a74fcd-37c0-4789-9414-9531b7e3f126 | External | None |
| 5a2712e9-70dc-4b0e-9281-17e02f4684c9 | management | None |
| 7aa697f5-0e60-4c15-b4cc-9cb659698512 | Internal | None |
+--------------------------------------+------------+------+
Output of print(networkId)
27a74fcd-37c0-4789-9414-9531b7e3f126
-
You should not recommend
os.popen()
in 2016. The Awk script could easily be replaced with native Python code.– tripleeeCommented Dec 3, 2018 at 5:49
Very simplest way to run any command and get the result back:
from commands import getstatusoutput
try:
return getstatusoutput("ls -ltr")
except Exception, e:
return None
-
4Indeed, the
commands
documentation from Python 2.7 says it was deprecated in 2.6 and will be removed in 3.0.– tripleeeCommented Dec 3, 2018 at 5:06
MOST OF THE CASES:
For the most of cases, a short snippet of code like this is all you are going to need:
import subprocess
import shlex
source = "test.txt"
destination = "test_copy.txt"
base = "cp {source} {destination}'"
cmd = base.format(source=source, destination=destination)
subprocess.check_call(shlex.split(cmd))
It is clean and simple.
subprocess.check_call
run command with arguments and wait for command to complete.
shlex.split
split the string cmd using shell-like syntax
REST OF THE CASES:
If this do not work for some specific command, most probably you have a problem with command-line interpreters. The operating system chose the default one which is not suitable for your type of program or could not found an adequate one on the system executable path.
Example:
Using the redirection operator on a Unix system
input_1 = "input_1.txt"
input_2 = "input_2.txt"
output = "merged.txt"
base_command = "/bin/bash -c 'cat {input} >> {output}'"
base_command.format(input_1, output=output)
subprocess.check_call(shlex.split(base_command))
base_command.format(input_2, output=output)
subprocess.check_call(shlex.split(base_command))
As it is stated in The Zen of Python: Explicit is better than implicit
So if using a Python >=3.6 function, it would look something like this:
import subprocess
import shlex
def run_command(cmd_interpreter: str, command: str) -> None:
base_command = f"{cmd_interpreter} -c '{command}'"
subprocess.check_call(shlex.split(base_command)
Here are my two cents: In my view, this is the best practice when dealing with external commands...
These are the return values from the execute method...
pass, stdout, stderr = execute(["ls","-la"],"/home/user/desktop")
This is the execute method...
def execute(cmdArray,workingDir):
stdout = ''
stderr = ''
try:
try:
process = subprocess.Popen(cmdArray,cwd=workingDir, stdout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize=1)
except OSError:
return [False, '', 'ERROR : command(' + ' '.join(cmdArray) + ') could not get executed!']
for line in iter(process.stdout.readline, b''):
try:
echoLine = line.decode("utf-8")
except:
echoLine = str(line)
stdout += echoLine
for line in iter(process.stderr.readline, b''):
try:
echoLine = line.decode("utf-8")
except:
echoLine = str(line)
stderr += echoLine
except (KeyboardInterrupt,SystemExit) as err:
return [False,'',str(err)]
process.stdout.close()
returnCode = process.wait()
if returnCode != 0 or stderr != '':
return [False, stdout, stderr]
else:
return [True, stdout, stderr]
-
1
-
Better yet, avoid
Popen()
and use the higher-level API which is now collected into the single functionsubprocess.run()
– tripleeeCommented Dec 3, 2018 at 5:27
Often, I use the following function for external commands, and this is especially handy for long running processes. The below method tails process output while it is running and returns the output, raises an exception if process fails.
It comes out if the process is done using the poll() method on the process.
import subprocess,sys
def exec_long_running_proc(command, args):
cmd = "{} {}".format(command, " ".join(str(arg) if ' ' not in arg else arg.replace(' ','\ ') for arg in args))
print(cmd)
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
# Poll process for new output until finished
while True:
nextline = process.stdout.readline().decode('UTF-8')
if nextline == '' and process.poll() is not None:
break
sys.stdout.write(nextline)
sys.stdout.flush()
output = process.communicate()[0]
exitCode = process.returncode
if (exitCode == 0):
return output
else:
raise Exception(command, exitCode, output)
You can invoke it like this:
exec_long_running_proc(command = "hive", args=["-f", hql_path])
-
1You'll get unexpected results passing an arg with space. Using
repr(arg)
instead ofstr(arg)
might help by the mere coincidence that python and sh escape quotes the same way– sbkCommented May 17, 2018 at 12:08 -
1@sbk
repr(arg)
didn't really help, the above code handles spaces as well. Now the following worksexec_long_running_proc(command = "ls", args=["-l", "~/test file*"])
– am5Commented Nov 17, 2018 at 0:07
Just to add to the discussion, if you include using a Python console, you can call external commands from IPython. While in the IPython prompt, you can call shell commands by prefixing '!'. You can also combine Python code with the shell, and assign the output of shell scripts to Python variables.
For instance:
In [9]: mylist = !ls
In [10]: mylist
Out[10]:
['file1',
'file2',
'file3',]
I wrote a small library to help with this use case:
https://pypi.org/project/citizenshell/
It can be installed using
pip install citizenshell
And then used as follows:
from citizenshell import sh
assert sh("echo Hello World") == "Hello World"
You can separate standard output from standard error and extract the exit code as follows:
result = sh(">&2 echo error && echo output && exit 13")
assert result.stdout() == ["output"]
assert result.stderr() == ["error"]
assert result.exit_code() == 13
And the cool thing is that you don't have to wait for the underlying shell to exit before starting processing the output:
for line in sh("for i in 1 2 3 4; do echo -n 'It is '; date +%H:%M:%S; sleep 1; done", wait=False)
print ">>>", line + "!"
will print the lines as they are available thanks to the wait=False
>>> It is 14:24:52!
>>> It is 14:24:53!
>>> It is 14:24:54!
>>> It is 14:24:55!
More examples can be found at https://github.com/meuter/citizenshell
Calling an external command in Python
A simple way to call an external command is using os.system(...)
. And this function returns the exit value of the command. But the drawback is we won't get stdout and stderr.
ret = os.system('some_cmd.sh')
if ret != 0 :
print 'some_cmd.sh execution returned failure'
Calling an external command in Python in background
subprocess.Popen
provides more flexibility for running an external command rather than using os.system
. We can start a command in the background and wait for it to finish. And after that we can get the stdout and stderr.
proc = subprocess.Popen(["./some_cmd.sh"], stdout=subprocess.PIPE)
print 'waiting for ' + str(proc.pid)
proc.wait()
print 'some_cmd.sh execution finished'
(out, err) = proc.communicate()
print 'some_cmd.sh output : ' + out
Calling a long running external command in Python in the background and stop after some time
We can even start a long running process in the background using subprocess.Popen
and kill it after sometime once its task is done.
proc = subprocess.Popen(["./some_long_run_cmd.sh"], stdout=subprocess.PIPE)
# Do something else
# Now some_long_run_cmd.sh exeuction is no longer needed, so kill it
os.system('kill -15 ' + str(proc.pid))
print 'Output : ' proc.communicate()[0]
Use:
import subprocess
p = subprocess.Popen("df -h", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0]
print p.split("\n")
It gives nice output which is easier to work with:
['Filesystem Size Used Avail Use% Mounted on',
'/dev/sda6 32G 21G 11G 67% /',
'none 4.0K 0 4.0K 0% /sys/fs/cgroup',
'udev 1.9G 4.0K 1.9G 1% /dev',
'tmpfs 387M 1.4M 386M 1% /run',
'none 5.0M 0 5.0M 0% /run/lock',
'none 1.9G 58M 1.9G 3% /run/shm',
'none 100M 32K 100M 1% /run/user',
'/dev/sda5 340G 222G 100G 69% /home',
'']
As an example (in Linux):
import subprocess
subprocess.run('mkdir test.dir', shell=True)
This creates test.dir in the current directory. Note that this also works:
import subprocess
subprocess.call('mkdir test.dir', shell=True)
The equivalent code using os.system is:
import os
os.system('mkdir test.dir')
Best practice would be to use subprocess instead of os, with .run favored over .call. All you need to know about subprocess is here. Also, note that all Python documentation is available for download from here. I downloaded the PDF packed as .zip. I mention this because there's a nice overview of the os module in tutorial.pdf (page 81). Besides, it's an authoritative resource for Python coders.
-
2According to docs.python.org/2/library/…, "shell=True" may raise a security concern.– NickCommented Mar 20, 2018 at 18:54
-
@Nick Predley: noted, but "shell=False" doesn't perform the desired function. What specifically are the security concerns and what's the alternative? Please let me know asap: I do not wish to post anything which may cause problems for anyone viewing this.– user8468899Commented Mar 21, 2018 at 19:49
-
1The basic warning is in the documentation but this question explains it in more detail: stackoverflow.com/questions/3172470/…– tripleeeCommented Dec 3, 2018 at 5:14
There are a lot of different ways to run external commands in Python, and all of them have their own plus sides and drawbacks.
My colleagues and me have been writing Python system administration tools, so we need to run a lot of external commands, and sometimes you want them to block or run asynchronously, time-out, update every second, etc.
There are also different ways of handling the return code and errors, and you might want to parse the output, and provide new input (in an expect kind of style). Or you will need to redirect standard input, standard output, and standard error to run in a different tty (e.g., when using GNU Screen).
So you will probably have to write a lot of wrappers around the external command. So here is a Python module which we have written which can handle almost anything you would want, and if not, it's very flexible so you can easily extend it:
https://github.com/hpcugent/vsc-base/blob/master/lib/vsc/utils/run.py
It doesn't work stand-alone and requires some of our other tools, and got a lot of specialised functionality over the years, so it might not be a drop-in replacement for you, but it can give you a lot of information on how the internals of Python for running commands work and ideas on how to handle certain situations.
For using subprocess
in Python 3.5+, the following did the trick for me on Linux:
import subprocess
# subprocess.run() returns a completed process object that can be inspected
c = subprocess.run(["ls", "-ltrh"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(c.stdout.decode('utf-8'))
As mentioned in the documentation, PIPE
values are byte sequences and for properly showing them decoding should be considered. For later versions of Python, text=True
and encoding='utf-8'
are added to kwargs of subprocess.run()
.
The output of the abovementioned code is:
total 113M
-rwxr-xr-x 1 farzad farzad 307 Jan 15 2018 vpnscript
-rwxrwxr-x 1 farzad farzad 204 Jan 15 2018 ex
drwxrwxr-x 4 farzad farzad 4.0K Jan 22 2018 scripts
.... # Some other lines
Here is calling an external command and return or print the command's output:
Python Subprocess check_output is good for
Run command with arguments and return its output as a byte string.
import subprocess
proc = subprocess.check_output('ipconfig /all')
print proc
-
The argument should properly be tokenized into a list, or you should explicitly pass in
shell=True
. In Python 3.x (where x > 3 I think) you can retrieve the output as a proper string withuniversal_newlines=True
and you probably want to switch tosubproces.run()
– tripleeeCommented Dec 3, 2018 at 5:22
If you need to call a shell command from a Python notebook (like Jupyter, Zeppelin, Databricks, or Google Cloud Datalab) you can just use the !
prefix.
For example,
!ls -ilF
Use subprocess.call:
from subprocess import call
# Using list
call(["echo", "Hello", "world"])
# Single string argument varies across platforms so better split it
call("echo Hello world".split(" "))
If you're writing a Python shell script and have IPython installed on your system, you can use the bang prefix to run a shell command inside IPython:
!ls
filelist = !ls
-
@PeterMortensen I don't think it works in DOS, but it should work in Cygwin. Commented Nov 30, 2019 at 1:39
After some research, I have the following code which works very well for me. It basically prints both standard output and standard error in real time.
stdout_result = 1
stderr_result = 1
def stdout_thread(pipe):
global stdout_result
while True:
out = pipe.stdout.read(1)
stdout_result = pipe.poll()
if out == '' and stdout_result is not None:
break
if out != '':
sys.stdout.write(out)
sys.stdout.flush()
def stderr_thread(pipe):
global stderr_result
while True:
err = pipe.stderr.read(1)
stderr_result = pipe.poll()
if err == '' and stderr_result is not None:
break
if err != '':
sys.stdout.write(err)
sys.stdout.flush()
def exec_command(command, cwd=None):
if cwd is not None:
print '[' + ' '.join(command) + '] in ' + cwd
else:
print '[' + ' '.join(command) + ']'
p = subprocess.Popen(
command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd
)
out_thread = threading.Thread(name='stdout_thread', target=stdout_thread, args=(p,))
err_thread = threading.Thread(name='stderr_thread', target=stderr_thread, args=(p,))
err_thread.start()
out_thread.start()
out_thread.join()
err_thread.join()
return stdout_result + stderr_result
-
4your code may lose data when the subprocess exits while there is some data is buffered. Read until EOF instead, see teed_call()– jfsCommented Jul 13, 2015 at 18:52
Update 2015: Python 3.5 added subprocess.run which is much easier to use than subprocess.Popen. I recommend that.
>>> subprocess.run(["ls", "-l"]) # doesn't capture output
CompletedProcess(args=['ls', '-l'], returncode=0)
>>> subprocess.run("exit 1", shell=True, check=True)
Traceback (most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
>>> subprocess.run(["ls", "-l", "/dev/null"], capture_output=True)
CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0,
stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n', stderr=b'')
-
10Deprecated doesn't only mean "isn't developed anymore" but also "you are discouraged from using this". Deprecated features may break anytime, may be removed anytime, or may dangerous. You should never use this in important code. Deprecation is merely a better way than removing a feature immediately, because it gives programmers the time to adapt and replace their deprecated functions.– MischCommented Apr 19, 2013 at 8:07
-
4Just to prove my point: "Deprecated since version 2.6: The commands module has been removed in Python 3. Use the subprocess module instead."– MischCommented Apr 19, 2013 at 8:14
-
It's not dangerous! The Python devs are careful only to break features between major releases (ie. between 2.x and 3.x). I've been using the commands module since 2004's Python 2.4. It works the same today in Python 2.7. Commented Apr 23, 2013 at 16:09
-
6With dangerous, I didn't mean that it may be removed anytime (that's a different problem), neither did I say that it is dangerous to use this specific module. However it may become dangerous if a security vulnerability is discovered but the module isn't further developed or maintained. (I don't want to say that this module is or isn't vulnerable to security issues, just talking about deprecated stuff in general)– MischCommented Apr 23, 2013 at 16:23
For Python 3.5+ it is recommended that you use the run function from the subprocess module. This returns a CompletedProcess
object, from which you can easily obtain the output as well as return code.
from subprocess import PIPE, run
command = ['echo', 'hello']
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True)
print(result.returncode, result.stdout, result.stderr)
-
3answer with run function was added in 2015 year. You repeated it. I think it was a reason of down vote Commented Mar 11, 2017 at 18:27
If you are not using user input in the commands, you can use this:
from os import getcwd
from subprocess import check_output
from shlex import quote
def sh(command):
return check_output(quote(command), shell=True, cwd=getcwd(), universal_newlines=True).strip()
And use it as
branch = sh('git rev-parse --abbrev-ref HEAD')
shell=True
will spawn a shell, so you can use pipe and such shell things sh('ps aux | grep python')
. This is very very handy for running hardcoded commands and processing its output. The universal_lines=True
make sure the output is returned in a string instead of binary.
cwd=getcwd()
will make sure that the command is run with the same working directory as the interpreter. This is handy for Git commands to work like the Git branch name example above.
Some recipes
- free memory in megabytes:
sh('free -m').split('\n')[1].split()[1]
- free space on / in percent
sh('df -m /').split('\n')[1].split()[4][0:-1]
- CPU load
sum(map(float, sh('ps -ef -o pcpu').split('\n')[1:])
But this isn't safe for user input, from the documentation:
Security Considerations
Unlike some other popen functions, this implementation will never implicitly call a system shell. This means that all characters, including shell metacharacters, can safely be passed to child processes. If the shell is invoked explicitly, via shell=True, it is the application’s responsibility to ensure that all whitespace and metacharacters are quoted appropriately to avoid shell injection vulnerabilities.
When using shell=True, the shlex.quote() function can be used to properly escape whitespace and shell metacharacters in strings that are going to be used to construct shell commands.
Even using the shlex.quote()
, it is good to keep a little paranoid when using user inputs on shell commands. One option is using a hardcoded command to take some generic output and filtering by user input. Anyway using shell=False
will make sure that only the exactly process that you want to execute will be executed or you get a No such file or directory
error.
Also there is some performance impact on shell=True
, from my tests it seems about 20% slower than shell=False
(the default).
In [50]: timeit("check_output('ls -l'.split(), universal_newlines=True)", number=1000, globals=globals())
Out[50]: 2.6801227919995654
In [51]: timeit("check_output('ls -l', universal_newlines=True, shell=True)", number=1000, globals=globals())
Out[51]: 3.243950183999914
You can try using os.system()
for running external commands.
Example:
import os
try:
os.system('ls')
pass
except:
print("Error running command")
pass
In the example, the script imports os
and tries to run the command listed in os.system()
. If the command was to fail then it would print "Error running command" without the script stopping due to the error.
And yes, it’s just that simple!
Using the Popen
function of the subprocess
Python module is the simplest way of running Linux commands. In that, the Popen.communicate()
function will give your commands output. For example
import subprocess
..
process = subprocess.Popen(..) # Pass command and arguments to the function
stdout, stderr = process.communicate() # Get command output and error
..
-
This is no longer true, and probably wasn't when this answer was posted. You should prefer
subprocess.check_call()
and friends unless you absolutely need the lower-level control of the more-complexPopen()
. In recent Python versions, the go-to workhorse issubprocess.run()
– tripleeeCommented Dec 3, 2018 at 5:30
There are many ways to call a command.
- For example:
if and.exe
needs two parameters. In cmd we can call sample.exe
use this:
and.exe 2 3
and it show 5
on screen.
If we use a Python script to call and.exe
, we should do like..
os.system(cmd,...)
os.system(("and.exe" + " " + "2" + " " + "3"))
os.popen(cmd,...)
os.popen(("and.exe" + " " + "2" + " " + "3"))
subprocess.Popen(cmd,...)
subprocess.Popen(("and.exe" + " " + "2" + " " + "3"))
It's too hard, so we can join cmd with a space:
import os
cmd = " ".join(exename,parameters)
os.popen(cmd)
-
os.popen
should not be recommended and perhaps even mentioned any longer. Thesubpocess
example should pass the arguments as a list instead of joining them with spaces.– tripleeeCommented Dec 3, 2018 at 5:25
import subprocess
p = subprocess.run(["ls", "-ltr"], capture_output=True)
print(p.stdout.decode(), p.stderr.decode())
-
An explanation would be in order. E.g, what is the idea and how is it different from the previous 50 answers 11 years later? Commented Apr 7, 2021 at 17:49
-
This is identical to your own answer from a few days earlier only with even less detail.– tripleeeCommented Jun 9, 2021 at 19:34
os.popen()
is the easiest and the most safest way to execute a command. You can execute any command that you run on the command line. In addition you will also be able to capture the output of the command using os.popen().read()
You can do it like this:
import os
output = os.popen('Your Command Here').read()
print (output)
An example where you list all the files in the current directory:
import os
output = os.popen('ls').read()
print (output)
# Outputs list of files in the directory
I would recommend the following method 'run' and it will help us in getting standard output, standard error and exit status as a dictionary; the caller of this can read the dictionary return by 'run' method to know the actual state of the process.
def run (cmd):
print "+ DEBUG exec({0})".format(cmd)
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True, shell=True)
(out, err) = p.communicate()
ret = p.wait()
out = filter(None, out.split('\n'))
err = filter(None, err.split('\n'))
ret = True if ret == 0 else False
return dict({'output': out, 'error': err, 'status': ret})
#end
-
1This incompletely reimplements something like
subprocess.run()
. You should particularly avoidshell=True
when it's not strictly necessary.– tripleeeCommented Dec 3, 2018 at 5:51 -
There are a number of ways of calling an external command from Python. There are some functions and modules with the good helper functions that can make it really easy. But the recommended thing among all is the subprocess
module.
import subprocess as s
s.call(["command.exe", "..."])
The call function will start the external process, pass some command line arguments and wait for it to finish. When it finishes you continue executing. Arguments in call function are passed through the list. The first argument in the list is the command typically in the form of an executable file and subsequent arguments in the list whatever you want to pass.
If you have called processes from the command line in the windows before, you'll be aware that you often need to quote arguments. You need to put quotations mark around it. If there's a space then there's a backslash and there are some complicated rules, but you can avoid a whole lot of that in Python by using subprocess
module because it is a list and each item is known to be a distinct and python can get quoting correctly for you.
In the end, after the list, there are a number of optional parameters one of these is a shell and if you set shell equals to true then your command is going to be run as if you have typed in at the command prompt.
s.call(["command.exe", "..."], shell=True)
This gives you access to functionality like piping, you can redirect to files, you can call multiple commands in one thing.
One more thing, if your script relies on the process succeeding then you want to check the result and the result can be checked with the check call helper function.
s.check_call(...)
It is exactly the same as a call function, it takes the same arguments, takes the same list, you can pass in any of the extra arguments but it going to wait for the functions to complete. And if the exit code of the function is anything other then zero, it will through an exception in the python script.
Finally, if you want tighter control Popen
constructor which is also from the subprocess
module. It also takes the same arguments as incall & check_call function but it returns an object representing the running process.
p=s.Popen("...")
It does not wait for the running process to finish also it's not going to throw any exception immediately but it gives you an object that will let you do things like wait for it to finish, let you communicate to it, you can redirect standard input, standard output if you want to display output somewhere else and a lot more.
You can run any command using Popen from the subprocess module.
from subprocess import Popen
First of all, a command object is created with all arguments which you want to run. For example, in the snippet below, the gunicorm command object has been formed with all the arguments:
cmd = (
"gunicorn "
"-c gunicorn_conf.py "
"-w {workers} "
"--timeout {timeout} "
"-b {address}:{port} "
"--limit-request-line 0 "
"--limit-request-field_size 0 "
"--log-level debug "
"--max-requests {max_requests} "
"manage:app").format(**locals())
Then this command object is used with Popen to instantiate a process:
process = Popen(cmd, shell=True)
This process can be terminated as well based upon any signal, using the code line below:
Popen.terminate(process)
And you can wait till the completion of above command's execution:
process.wait()
Here there are a lot of answers, but none fulfilled all my needs.
- I need to run the command and capture the output and exit code.
- I need to timeout the executed program and force it to exit if timeout is reached, and kill all its child processes.
- and I need that it works in Windows XP and later, Cygwin and Linux. In Python 2 and 3.
So I created this:
def _run(command, timeout_s=False, shell=False):
### run a process, capture the output and wait for it to finish. if timeout is specified then Kill the subprocess and its children when the timeout is reached (if parent did not detach)
## usage: _run(arg1, arg2, arg3)
# arg1: command + arguments. Always pass a string; the function will split it when needed
# arg2: (optional) timeout in seconds before force killing
# arg3: (optional) shell usage. default shell=False
## return: a list containing: exit code, output, and if timeout was reached or not
# - Tested on Python 2 and 3 on Windows XP, Windows 7, Cygwin and Linux.
# - preexec_fn=os.setsid (py2) is equivalent to start_new_session (py3) (works on Linux only), in Windows and Cygwin we use TASKKILL
# - we use stderr=subprocess.STDOUT to merge standard error and standard output
import sys, subprocess, os, signal, shlex, time
def _runPY3(command, timeout_s=None, shell=False):
# py3.3+ because: timeout was added to communicate() in py3.3.
new_session=False
if sys.platform.startswith('linux'): new_session=True
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, start_new_session=new_session, shell=shell)
try:
out = p.communicate(timeout=timeout_s)[0].decode('utf-8')
is_timeout_reached = False
except subprocess.TimeoutExpired:
print('Timeout reached: Killing the whole process group...')
killAll(p.pid)
out = p.communicate()[0].decode('utf-8')
is_timeout_reached = True
return p.returncode, out, is_timeout_reached
def _runPY2(command, timeout_s=0, shell=False):
preexec=None
if sys.platform.startswith('linux'): preexec=os.setsid
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, preexec_fn=preexec, shell=shell)
start_time = time.time()
is_timeout_reached = False
while timeout_s and p.poll() == None:
if time.time()-start_time >= timeout_s:
print('Timeout reached: Killing the whole process group...')
killAll(p.pid)
is_timeout_reached = True
break
time.sleep(1)
out = p.communicate()[0].decode('utf-8')
return p.returncode, out, is_timeout_reached
def killAll(ParentPid):
if sys.platform.startswith('linux'):
os.killpg(os.getpgid(ParentPid), signal.SIGTERM)
elif sys.platform.startswith('cygwin'):
# subprocess.Popen(shlex.split('bash -c "TASKKILL /F /PID $(</proc/{pid}/winpid) /T"'.format(pid=ParentPid)))
winpid=int(open("/proc/{pid}/winpid".format(pid=ParentPid)).read())
subprocess.Popen(['TASKKILL', '/F', '/PID', str(winpid), '/T'])
elif sys.platform.startswith('win32'):
subprocess.Popen(['TASKKILL', '/F', '/PID', str(ParentPid), '/T'])
# - In Windows, we never need to split the command, but in Cygwin and Linux we need to split if shell=False (default), shlex will split the command for us
if shell==False and (sys.platform.startswith('cygwin') or sys.platform.startswith('linux')):
command=shlex.split(command)
if sys.version_info >= (3, 3): # py3.3+
if timeout_s==False:
returnCode, output, is_timeout_reached = _runPY3(command, timeout_s=None, shell=shell)
else:
returnCode, output, is_timeout_reached = _runPY3(command, timeout_s=timeout_s, shell=shell)
else: # Python 2 and up to 3.2
if timeout_s==False:
returnCode, output, is_timeout_reached = _runPY2(command, timeout_s=0, shell=shell)
else:
returnCode, output, is_timeout_reached = _runPY2(command, timeout_s=timeout_s, shell=shell)
return returnCode, output, is_timeout_reached
Then use it like this:
Always pass the command as one string (it is easier). You do not need to split it; the function will split it when needed.
If your command works in your shell, it will work with this function, so test your command in your shell first cmd/Bash.
So we can use it like this with a timeout:
a=_run('cmd /c echo 11111 & echo 22222 & calc',3)
for i in a[1].splitlines(): print(i)
Or without a timeout:
b=_run('cmd /c echo 11111 & echo 22222 & calc')
More examples:
b=_run('''wmic nic where 'NetConnectionID="Local Area Connection"' get NetConnectionStatus /value''')
print(b)
c=_run('cmd /C netsh interface ip show address "Local Area Connection"')
print(c)
d=_run('printf "<%s>\n" "{foo}"')
print(d)
You can also specify shell=True, but it is useless in most cases with this function. I prefer to choose myself the shell I want, but here it is if you need it too:
# windows
e=_run('echo 11111 & echo 22222 & calc',3, shell=True)
print(e)
# Cygwin/Linux:
f=_run('printf "<%s>\n" "{foo}"', shell=True)
print(f)
Why did I not use the simpler new method subprocess.run()
?
- because it is supported in Python 3.7+, but the last supported Python version in Windows XP is 3.4.
- and because the timeout argument of this function is useless in Windows, it does not kill the child processes of the executed command.
- if you use the
capture_output
+timeout
argument, it will hang if there is a child process still running. And it is still broken in Windows, for which the issue 31447 is still open.
-
1I removed the "2022 answer" because it’s misleading. In 2022 you don’t support winxp nor Python 2 and you use
subprocess.run()
. The question is not "how do I run system commands on old platforms" but "how do I run system commands", period. Commented May 24, 2022 at 10:08