348

If I do the following:

import subprocess
from cStringIO import StringIO
subprocess.Popen(['grep','f'],stdout=subprocess.PIPE,stdin=StringIO('one\ntwo\nthree\nfour\nfive\nsix\n')).communicate()[0]

I get:

Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/build/toolchain/mac32/python-2.4.3/lib/python2.4/subprocess.py", line 533, in __init__
    (p2cread, p2cwrite,
  File "/build/toolchain/mac32/python-2.4.3/lib/python2.4/subprocess.py", line 830, in _get_handles
    p2cread = stdin.fileno()
AttributeError: 'cStringIO.StringI' object has no attribute 'fileno'

Apparently a cStringIO.StringIO object doesn't quack close enough to a file duck to suit subprocess.Popen. How do I work around this?

3

12 Answers 12

397

Popen.communicate() documentation:

Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.

Replacing os.popen*

    pipe = os.popen(cmd, 'w', bufsize)
    # ==>
    pipe = Popen(cmd, shell=True, bufsize=bufsize, stdin=PIPE).stdin

Warning Use communicate() rather than stdin.write(), stdout.read() or stderr.read() to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

So your example could be written as follows:

from subprocess import Popen, PIPE, STDOUT

p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)    
grep_stdout = p.communicate(input=b'one\ntwo\nthree\nfour\nfive\nsix\n')[0]
print(grep_stdout.decode())
# -> four
# -> five
# ->

On Python 3.5+ (3.6+ for encoding), you could use subprocess.run, to pass input as a string to an external command and get its exit status, and its output as a string back in one call:

#!/usr/bin/env python3
from subprocess import run, PIPE

p = run(['grep', 'f'], stdout=PIPE,
        input='one\ntwo\nthree\nfour\nfive\nsix\n', encoding='ascii')
print(p.returncode)
# -> 0
print(p.stdout)
# -> four
# -> five
# -> 
14
  • 17
    This is NOT a good solution. In particular, you cannot asynchronously process p.stdout.readline output if you do this since you'd have to wait for the entire stdout to arrive. It's is also memory-inefficient.
    – OTZ
    Aug 20, 2010 at 21:59
  • 10
    @OTZ What's a better solution?
    – Nick T
    Nov 17, 2010 at 21:27
  • 13
    @Nick T: "better" depends on context. Newton's laws are good for the domain they are applicable but you need special relativity to design GPS. See Non-blocking read on a subprocess.PIPE in python.
    – jfs
    Oct 18, 2011 at 20:25
  • 9
    But note the NOTE for communicate: "do not use this method if the data size is large or unlimited"
    – Owen
    Jan 22, 2014 at 0:21
  • 2
    You need python 3.6 to use the input arg with subprocess.run(). Older versions of python3 work if you do this: p = run(['grep', 'f'], stdout=PIPE, input=some_string.encode('ascii'))
    – TaborKelly
    Aug 3, 2018 at 19:27
50

I figured out this workaround:

>>> p = subprocess.Popen(['grep','f'],stdout=subprocess.PIPE,stdin=subprocess.PIPE)
>>> p.stdin.write(b'one\ntwo\nthree\nfour\nfive\nsix\n') #expects a bytes type object
>>> p.communicate()[0]
'four\nfive\n'
>>> p.stdin.close()

Is there a better one?

6
  • 27
    @Moe: stdin.write() usage is discouraged, p.communicate() should be used. See my answer.
    – jfs
    Oct 3, 2008 at 14:26
  • 12
    Per the subprocess documentation: Warning - Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
    – Jason Mock
    Aug 25, 2010 at 20:49
  • 2
    I think this is good way to do it if you're confident that your stdout/err won't ever fill up (for instance, it's going to a file, or another thread is eating it) and you have an unbounded amount of data to be sent to stdin.
    – Lucretiel
    May 9, 2016 at 19:24
  • 1
    In particular, doing it this way still ensures that stdin is closed, so that if the subprocesses is one that consumes input forever, the communicate will close the pipe and allow the process to end gracefully.
    – Lucretiel
    May 9, 2016 at 19:25
  • @Lucretiel, if the process consumes stdin forever, then presumably it can still write stdout forever, so we'd need completely different techniques all-round (can't read() from it, as communicate() does even with no arguments). Mar 13, 2020 at 11:27
42

There's a beautiful solution if you're using Python 3.4 or better. Use the input argument instead of the stdin argument, which accepts a bytes argument:

output_bytes = subprocess.check_output(
    ["sed", "s/foo/bar/"],
    input=b"foo",
)

This works for check_output and run, but not call or check_call for some reason.

In Python 3.7+, you can also add text=True to make check_output take a string as input and return a string (instead of bytes):

output_string = subprocess.check_output(
    ["sed", "s/foo/bar/"],
    input="foo",
    text=True,
)
3
  • 7
    @vidstige You're right, that's weird. I would consider filing this as an Python bug, I don't see any good reason in why check_output should have an input argument, but not call.
    – Flimm
    Oct 5, 2017 at 14:35
  • 3
    This is the best answer for Python 3.4+ (using it in Python 3.6). It indeed does not work with check_call but it works for run. It also works with input=string as long as you pass an encoding argument too according to the documentation. Feb 22, 2019 at 2:48
  • @Flimm the reason is obvious: run and check_output use communicate under the hood, while call and check_call do not. communicate is much more heavy, since it involves select to work with streams, call and check_call are much simplier and faster.
    – Vadim Fint
    Nov 12, 2022 at 15:12
32

I'm a bit surprised nobody suggested creating a pipe, which is in my opinion the far simplest way to pass a string to stdin of a subprocess:

read, write = os.pipe()
os.write(write, "stdin input here")
os.close(write)

subprocess.check_call(['your-command'], stdin=read)
6
  • 3
    The os and the subprocess documentation both agree that you should prefer the latter over the former. This is a legacy solution which has a (slightly less concise) standard replacement; the accepted answer quotes the pertinent documentation.
    – tripleee
    May 4, 2016 at 11:53
  • 1
    I'm not sure that's correct, tripleee. The quoted documentation says why it is hard to use the pipes created by the process, but in this solution it creates a pipe and passes it in. I believe it avoids the potential deadlock problems of managing the pipes after the process has already started. May 7, 2016 at 15:09
  • os.popen is deprecated in favour of subprocess
    – hd1
    Feb 18, 2017 at 17:30
  • 3
    -1: it leads to the deadlock, it may loose data. This functionality is already provided by the subprocess module. Use it instead of reimplementing it poorly (try to write a value that is larger than an OS pipe buffer)
    – jfs
    Aug 18, 2017 at 6:24
  • 2
    @tripleee the implementation of pipes in subprocess module is laughably bad, and is impossible to control. You cannot even get the information about the size of the built-in buffer, not to mention, you cannot tell it what are the read and write ends of the pipe, nor can you change the built-in buffer. In not so many words: subprocess pipes are trash. Don't use them.
    – wvxvw
    Dec 27, 2020 at 13:18
15

I am using python3 and found out that you need to encode your string before you can pass it into stdin:

p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=PIPE)
out, err = p.communicate(input='one\ntwo\nthree\nfour\nfive\nsix\n'.encode())
print(out)
3
  • 5
    You don't specifically need to encode the input, it just wants a bytes-like object (e.g. b'something'). It will return err and out as bytes also. If you want to avoid this, you can pass universal_newlines=True to Popen. Then it will accept input as str and will return err/out as str also.
    – Six
    Jan 17, 2016 at 15:00
  • 2
    But beware, universal_newlines=True will also convert your newlines to match your system
    – Nacht
    Feb 25, 2016 at 12:35
  • 1
    If you're using Python 3, see my answer for an even more convenient solution.
    – Flimm
    Dec 8, 2016 at 10:05
12

Apparently a cStringIO.StringIO object doesn't quack close enough to a file duck to suit subprocess.Popen

I'm afraid not. The pipe is a low-level OS concept, so it absolutely requires a file object that is represented by an OS-level file descriptor. Your workaround is the right one.

11
from subprocess import Popen, PIPE
from tempfile import SpooledTemporaryFile as tempfile
f = tempfile()
f.write('one\ntwo\nthree\nfour\nfive\nsix\n')
f.seek(0)
print Popen(['/bin/grep','f'],stdout=PIPE,stdin=f).stdout.read()
f.close()
1
  • 3
    fyi, tempfile.SpooledTemporaryFile.__doc__ says: Temporary file wrapper, specialized to switch from StringIO to a real file when it exceeds a certain size or when a fileno is needed.
    – Doug F
    Aug 9, 2013 at 20:18
7
"""
Ex: Dialog (2-way) with a Popen()
"""

p = subprocess.Popen('Your Command Here',
                 stdout=subprocess.PIPE,
                 stderr=subprocess.STDOUT,
                 stdin=PIPE,
                 shell=True,
                 bufsize=0)
p.stdin.write('START\n')
out = p.stdout.readline()
while out:
  line = out
  line = line.rstrip("\n")

  if "WHATEVER1" in line:
      pr = 1
      p.stdin.write('DO 1\n')
      out = p.stdout.readline()
      continue

  if "WHATEVER2" in line:
      pr = 2
      p.stdin.write('DO 2\n')
      out = p.stdout.readline()
      continue
"""
..........
"""

out = p.stdout.readline()

p.wait()
1
  • 5
    Because shell=True is so commonly used for no good reason, and this is a popular question, let me point out that there are a lot of situations where Popen(['cmd', 'with', 'args']) is decidedly better than Popen('cmd with args', shell=True) and having the shell break the command and arguments into tokens, but not otherwise providing anything useful, while adding a significant amount of complexity and thus also attack surface.
    – tripleee
    Oct 29, 2014 at 13:43
7

On Python 3.7+ do this:

my_data = "whatever you want\nshould match this f"
subprocess.run(["grep", "f"], text=True, input=my_data)

and you'll probably want to add capture_output=True to get the output of running the command as a string.

On older versions of Python, replace text=True with universal_newlines=True:

subprocess.run(["grep", "f"], universal_newlines=True, input=my_data)
0
6

Beware that Popen.communicate(input=s)may give you trouble ifsis too big, because apparently the parent process will buffer it before forking the child subprocess, meaning it needs "twice as much" used memory at that point (at least according to the "under the hood" explanation and linked documentation found here). In my particular case,swas a generator that was first fully expanded and only then written tostdin so the parent process was huge right before the child was spawned, and no memory was left to fork it:

File "/opt/local/stow/python-2.7.2/lib/python2.7/subprocess.py", line 1130, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory

3

This is overkill for grep, but through my journeys I've learned about the Linux command expect, and the python library pexpect

  • expect: dialogue with interactive programs
  • pexpect: Python module for spawning child applications; controlling them; and responding to expected patterns in their output.
import pexpect
child = pexpect.spawn('grep f', timeout=10)
child.sendline('text to match')
print(child.before)

Working with interactive shell applications like ftp is trivial with pexpect

import pexpect
child = pexpect.spawn ('ftp ftp.openbsd.org')
child.expect ('Name .*: ')
child.sendline ('anonymous')
child.expect ('Password:')
child.sendline ('[email protected]')
child.expect ('ftp> ')
child.sendline ('ls /pub/OpenBSD/')
child.expect ('ftp> ')
print child.before   # Print the result of the ls command.
child.interact()     # Give control of the child to the user.
2
p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)    
p.stdin.write('one\n')
time.sleep(0.5)
p.stdin.write('two\n')
time.sleep(0.5)
p.stdin.write('three\n')
time.sleep(0.5)
testresult = p.communicate()[0]
time.sleep(0.5)
print(testresult)
1
  • NameError: global name 'PIPE' is not defined Apr 11, 2023 at 0:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.