38

How do I run a command with a pipe | in it?

The subprocess module seems complex...

Is there something like

output,error = `ps cax | grep something`

as in shell script?

1
  • Don't do this with subprocess. It's much easier to do this with the shell. Indeed, this is the one thing the shell does best.
    – S.Lott
    Jul 21, 2011 at 17:19

5 Answers 5

71

See Replacing shell pipeline:

import subprocess

proc1 = subprocess.Popen(['ps', 'cax'], stdout=subprocess.PIPE)
proc2 = subprocess.Popen(['grep', 'python'], stdin=proc1.stdout,
                         stdout=subprocess.PIPE, stderr=subprocess.PIPE)

proc1.stdout.close() # Allow proc1 to receive a SIGPIPE if proc2 exits.
out, err = proc2.communicate()
print('out: {0}'.format(out))
print('err: {0}'.format(err))

PS. Using shell=True can be dangerous. See for example the warning in the docs.


There is also the sh module which can make subprocess scripting in Python a lot more pleasant:

import sh
print(sh.grep(sh.ps("cax"), 'something'))
7
  • 2
    Great reference to the sh module, I highly recommend it. Dec 18, 2015 at 12:46
  • 3
    sh.grep(sh.ps('aux', _piped=True), 'something') - worked for me
    – Ben Usman
    Jun 3, 2016 at 5:42
  • @unutbu, Could you explain more detail why proc1.stdout.close() should be called?
    – SangminKim
    Dec 27, 2017 at 18:45
  • 1
    @SangminKim: This appears to have been fixed or worked-around in Python3, but in Python2.7 at least, if you were to spawn a long-running process such as find / -print in proc1 without calling proc1.stdout.close() and spawn a short-running command like head in proc2, then you would see find / -print is still running long after the call to head has ended. Including proc1.stdout.close() allows find to end soon after head finishes.
    – unutbu
    Dec 28, 2017 at 2:15
  • 1
    For more on SIGPIPE, see this comment, this post and this one too.
    – unutbu
    Dec 28, 2017 at 2:15
24

You've already accepted an answer, but:

Do you really need to use grep? I'd write something like:

import subprocess
ps = subprocess.Popen(('ps', 'cax'), stdout=subprocess.PIPE)
output = ps.communicate()[0]
for line in output.split('\n'):
    if 'something' in line:
        ...

This has the advantages of not involving shell=True and its riskiness, doesn't fork off a separate grep process, and looks an awful lot like the kind of Python you'd write to process data file-like objects.

1
  • 1
    for me, output was a [byte] and had to use it like str(output).split("\\n") Nov 4, 2020 at 9:12
12
import subprocess

process = subprocess.Popen("ps cax | grep something",
                             shell=True,
                             stdout=subprocess.PIPE,
                           )
stdout_list = process.communicate()[0].split('\n')
3
  • something in grep something would be a variable, should I construct the "ps cax | grep something" string with variable to run it? or is there other way of doing it?
    – eugene
    Jul 21, 2011 at 17:23
  • 1
  • 3
    @Eugene: You can construct the string using a variable, but be careful about where the variable is coming from. I.e. make sure it's not from a user who could make "something" into "something; rm -rf /". Building expressions to run with shell=True is a possible security risk.
    – Thomas K
    Jul 21, 2011 at 17:39
6

Drop that 'ps' subprocess and back away slowly! :)

Use the psutil module instead.

3
  • 8
    I want to use psutil, but this answer isn't very helpful without showing how it would be done.
    – user513951
    Aug 1, 2017 at 0:43
  • pls share example
    – Pradip Das
    Apr 12, 2018 at 4:19
  • 2
    import psutil; for proc in psutil.process_iter(): cmdline = " ".join(proc.cmdline()); if something in cmdline: break Sep 10, 2018 at 20:44
4
import os

os.system('ps -cax|grep something')

If you wanna replace grep argument with some variable:

os.system('ps -cax|grep '+your_var)
3
  • @Kirk Strauser, can't you get the output? I have testd it both in interactive python interpreter and running it as script, it works as expected(my python version is 2.7.1,if it helps)
    – Mark Ma
    Jul 22, 2011 at 13:01
  • 4
    os.system only returns the int exit code of the subprocess. If you're running it at an interactive prompt and seeing the output of ps, it's because ps is writing to stout. Python isn't actually capturing that output. Try it yourself: run a = os.system('ls'). You'll still see the output of ls, and a will be 0 (assuming ls didn't fail for some reason). Jul 22, 2011 at 16:05
  • This is unsafe. If your_var is set by someone else, this code lets them run any command they want. For example your_var = "a; echo you got hacked" Dec 27, 2019 at 4:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.