425

I'm trying to make a system call in Python and store the output to a string that I can manipulate in the Python program.

#!/usr/bin/python
import subprocess
p2 = subprocess.Popen("ntpq -p")

I've tried a few things including some of the suggestions here:

Retrieving the output of subprocess.call()

but without any luck.

1
  • 3
    It is good always to post actual code you ran and the actual traceback or unexpected bahaviour for concrete questions like this. For example, I do not know what you tried to do to get the output and I suspect you didn't actually get that far to start with—you would have gotten an error about not finding the file for "ntpq -p", which is a different part of the problem than you're asking about. Mar 23, 2010 at 19:39

15 Answers 15

592

In Python 2.7 or Python 3

Instead of making a Popen object directly, you can use the subprocess.check_output() function to store output of a command in a string:

from subprocess import check_output
out = check_output(["ntpq", "-p"])

In Python 2.4-2.6

Use the communicate method.

import subprocess
p = subprocess.Popen(["ntpq", "-p"], stdout=subprocess.PIPE)
out, err = p.communicate()

out is what you want.

Important note about the other answers

Note how I passed in the command. The "ntpq -p" example brings up another matter. Since Popen does not invoke the shell, you would use a list of the command and options—["ntpq", "-p"].

20
  • 4
    In this case, does python wait for this system call to finish? Or is it necessary to explicitly call the wait/waitpid function?
    – NoneType
    Jul 22, 2010 at 11:35
  • 10
    @NoneType, Popen.communicate does not return until the process has terminated. Jul 23, 2010 at 2:34
  • 12
    if you want to get error stream add stderr: p = subprocess.Popen(["ntpq", "-p"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    – Timofey
    May 27, 2014 at 12:29
  • 137
    Careful, subprocess.check_output() returns a bytes object, not a str. If all you want to do is print the result, that won't make any difference. But if you want to use a method such as myString.split("\n") on the result, you'll have to first decode the bytes object: subprocess.check_output(myParams).decode("utf-8") for instance.
    – TanguyP
    Sep 29, 2015 at 15:57
  • 35
    adding universal_newlines=True as a parameter helped me in Python 3 to get the string object. If universal_newlines is True, they are opened in text mode with default encoding. Otherwise, they are opened as binary streams. May 31, 2017 at 5:22
53

This worked for me for redirecting stdout (stderr can be handled similarly):

from subprocess import Popen, PIPE
pipe = Popen(path, stdout=PIPE)
text = pipe.communicate()[0]

If it doesn't work for you, please specify exactly the problem you're having.

3
  • 7
    This does produce some weird object. When I convert it to string, it escapes whitespace like \n. Nov 26, 2016 at 18:13
  • 2
    Note that this doesn't check if the subprocess ran correctly. You probably want to check that pipe.returncode == 0 after the last line too.
    – thakis
    Jun 8, 2018 at 14:56
  • 4
    the reason this works is because Popen returns a tuple of stdout and stderr, so when you access [0] you're just grabbing the stdout. You could also do text, err = pipe.communicate() and then text will have what you expect
    – Jona
    Oct 29, 2019 at 19:27
52

In Python 3.7+ you can use the new capture_output= keyword argument for subprocess.run:

import subprocess

p = subprocess.run(["echo", "hello world!"], capture_output=True, text=True)
assert p.stdout == 'hello world!\n'
47

Python 2: http://docs.python.org/2/library/subprocess.html#subprocess.Popen

from subprocess import PIPE, Popen

command = "ntpq -p"
process = Popen(command, stdout=PIPE, stderr=None, shell=True)
output = process.communicate()[0]
print output

In the Popen constructor, if shell is True, you should pass the command as a string rather than as a sequence. Otherwise, just split the command into a list:

command = ["ntpq", "-p"]
process = Popen(command, stdout=PIPE, stderr=None)

If you need to read also the standard error, into the Popen initialization, you should set stderr to PIPE or STDOUT:

command = "ntpq -p"
process = subprocess.Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
output, error = process.communicate()

NOTE: Starting from Python 2.7, you could/should take advantage of subprocess.check_output (https://docs.python.org/2/library/subprocess.html#subprocess.check_output).


Python 3: https://docs.python.org/3/library/subprocess.html#subprocess.Popen

from subprocess import PIPE, Popen

command = "ntpq -p"
with Popen(command, stdout=PIPE, stderr=None, shell=True) as process:
    output = process.communicate()[0].decode("utf-8")
    print(output)

NOTE: If you're targeting only versions of Python higher or equal than 3.5, then you could/should take advantage of subprocess.run (https://docs.python.org/3/library/subprocess.html#subprocess.run).

9
  • awesome this is what i was looking for
    – kRazzy R
    Nov 14, 2019 at 14:10
  • Probably also notice the place in the documentation where it says "don't use Popen() if you can use the higher-level API functions". You want check_output or in Python 3.4+ subprocess.run()
    – tripleee
    Jun 24, 2021 at 7:20
  • @tripleee the original question explicitly asks about "Store output of subprocess.Popen call in a string"... in any case, yes, according to the documentation the recommended approach is to "use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly". However, take in mind that run() was added starting from Python 3.5. Jun 25, 2021 at 13:58
  • For completeness, I've added to my answer a final note about this. ;) Jun 25, 2021 at 14:15
  • Is is not recommended to use shell=True
    – alper
    Aug 9, 2021 at 12:33
26

Assuming that pwd is just an example, this is how you can do it:

import subprocess

p = subprocess.Popen("pwd", stdout=subprocess.PIPE)
result = p.communicate()[0]
print result

See the subprocess documentation for another example and more information.

18

for Python 2.7+ the idiomatic answer is to use subprocess.check_output()

You should also note the handling of arguments when invoking a subprocess, as it can be a little confusing....

If args is just single command with no args of its own (or you have shell=True set), it can be a string. Otherwise it must be a list.

for example... to invoke the ls command, this is fine:

from subprocess import check_call
check_call('ls')

so is this:

from subprocess import check_call
check_call(['ls',])

however, if you want to pass some args to the shell command, you can't do this:

from subprocess import check_call
check_call('ls -al')

instead, you must pass it as a list:

from subprocess import check_call
check_call(['ls', '-al'])

the shlex.split() function can sometimes be useful to split a string into shell-like syntax before creating a subprocesses... like this:

from subprocess import check_call
import shlex
check_call(shlex.split('ls -al'))
1
  • 1
    Five years later, this question is still getting lots of love. Thanks for the 2.7+ update, Corey!
    – Mark
    Mar 11, 2016 at 8:06
11

This works perfectly for me:

import subprocess
try:
    #prints results and merges stdout and std
    result = subprocess.check_output("echo %USERNAME%", stderr=subprocess.STDOUT, shell=True)
    print result
    #causes error and merges stdout and stderr
    result = subprocess.check_output("copy testfds", stderr=subprocess.STDOUT, shell=True)
except subprocess.CalledProcessError, ex: # error code <> 0 
    print "--------error------"
    print ex.cmd
    print ex.message
    print ex.returncode
    print ex.output # contains stdout and stderr together 
10

This was perfect for me. You will get the return code, stdout and stderr in a tuple.

from subprocess import Popen, PIPE

def console(cmd):
    p = Popen(cmd, shell=True, stdout=PIPE)
    out, err = p.communicate()
    return (p.returncode, out, err)

For Example:

result = console('ls -l')
print 'returncode: %s' % result[0]
print 'output: %s' % result[1]
print 'error: %s' % result[2]
10

The accepted answer is still good, just a few remarks on newer features. Since python 3.6, you can handle encoding directly in check_output, see documentation. This returns a string object now:

import subprocess 
out = subprocess.check_output(["ls", "-l"], encoding="utf-8")

In python 3.7, a parameter capture_output was added to subprocess.run(), which does some of the Popen/PIPE handling for us, see the python docs :

import subprocess 
p2 = subprocess.run(["ls", "-l"], capture_output=True, encoding="utf-8")
p2.stdout
5

I wrote a little function based on the other answers here:

def pexec(*args):
    return subprocess.Popen(args, stdout=subprocess.PIPE).communicate()[0].rstrip()

Usage:

changeset = pexec('hg','id','--id')
branch = pexec('hg','id','--branch')
revnum = pexec('hg','id','--num')
print('%s : %s (%s)' % (revnum, changeset, branch))
3
 import os   
 list = os.popen('pwd').read()

In this case you will only have one element in the list.

2
  • 7
    os.popen is deprecated in favour of the subprocess module. Mar 23, 2010 at 19:17
  • 3
    This was quite helpful for admin on an old box using the 2.2.X series of Python.
    – Neil McF
    Apr 11, 2011 at 13:21
2
import subprocess
output = str(subprocess.Popen("ntpq -p",shell = True,stdout = subprocess.PIPE, 
stderr = subprocess.STDOUT).communicate()[0])

This is one line solution

1

The following captures stdout and stderr of the process in a single variable. It is Python 2 and 3 compatible:

from subprocess import check_output, CalledProcessError, STDOUT

command = ["ls", "-l"]
try:
    output = check_output(command, stderr=STDOUT).decode()
    success = True 
except CalledProcessError as e:
    output = e.output.decode()
    success = False

If your command is a string rather than an array, prefix this with:

import shlex
command = shlex.split(command)
1
  • Didn't know about shlex in stdlib :o
    – Jean Monet
    Jul 14, 2022 at 20:19
0

Use check_output method of subprocess module

import subprocess

address = '192.168.x.x'
res = subprocess.check_output(['ping', address, '-c', '3'])

Finally parse the string

for line in res.splitlines():

Hope it helps, happy coding

-1

For python 3.5 I put up function based on previous answer. Log may be removed, thought it's nice to have

import shlex
from subprocess import check_output, CalledProcessError, STDOUT


def cmdline(command):
    log("cmdline:{}".format(command))
    cmdArr = shlex.split(command)
    try:
        output = check_output(cmdArr,  stderr=STDOUT).decode()
        log("Success:{}".format(output))
    except (CalledProcessError) as e:
        output = e.output.decode()
        log("Fail:{}".format(output))
    except (Exception) as e:
        output = str(e);
        log("Fail:{}".format(e))
    return str(output)


def log(msg):
    msg = str(msg)
    d_date = datetime.datetime.now()
    now = str(d_date.strftime("%Y-%m-%d %H:%M:%S"))
    print(now + " " + msg)
    if ("LOG_FILE" in globals()):
        with open(LOG_FILE, "a") as myfile:
            myfile.write(now + " " + msg + "\n")

Not the answer you're looking for? Browse other questions tagged or ask your own question.