140

What I do in the command line:

cat file1 file2 file3 > myfile

What I want to do with python:

import subprocess, shlex
my_cmd = 'cat file1 file2 file3 > myfile'
args = shlex.split(my_cmd)
subprocess.call(args) # spits the output in the window i call my python program
4
  • Executing such a command in subprocess would not give you any output. May be you want to run it without > myfile redirecting output from cat file1 file2 file3 into python?
    – PoltoS
    Commented Feb 11, 2011 at 2:52
  • @PoltoS I want to join some files and then process the resulting file. I thought using cat was the easiest alternative. Is there a better/pythonic way to do it? Commented Feb 11, 2011 at 2:55
  • os.sendfile()-based solution is possible, see Reproduce the unix cat command in python
    – jfs
    Commented May 12, 2015 at 18:12
  • 1
    I think that output redirection ('>' or '>>') doesn't work in subprocess.Popen (at least in Python 2.7) (in shell=True mode) In this example, as others point out, you can work around this by not using redirection, but in other cases redirection is useful. If redirection or piping is not supported in subprocess.Popen is should be documented (and/or os.system() should be not be deprecated until this is fixed)
    – Ribo
    Commented Jul 19, 2016 at 12:50

6 Answers 6

380

In Python 3.5+ to redirect the output, just pass an open file handle for the stdout argument to subprocess.run:

# Use a list of args instead of a string
input_files = ['file1', 'file2', 'file3']
my_cmd = ['cat'] + input_files
with open('myfile', "w") as outfile:
    subprocess.run(my_cmd, stdout=outfile)

As others have pointed out, the use of an external command like cat for this purpose is completely extraneous.

9
  • 9
    This should be the answer for the general question of piping when using the shell from Python Commented Jul 28, 2014 at 20:38
  • 66
    This is the correct answer, not the one marked as correct. Commented Oct 3, 2014 at 20:29
  • 10
    For Python 3.5+ use subprocess.run(my_cmd, stdout=outfile) which is replacing subprocess.call(...) Commented May 18, 2018 at 13:53
  • 2
    Of note, this does not work with custom file objects, if they don't have a fileno field (if they are not a real file.) Commented Jun 21, 2018 at 19:45
  • 1
    Can I do an append (>>) with subprocess.call? [edit: apparently yes, replacing "w" with "a"]
    – Cœur
    Commented Nov 11, 2019 at 12:06
17

UPDATE: os.system is discouraged, albeit still available in Python 3.


Use os.system:

os.system(my_cmd)

If you really want to use subprocess, here's the solution (mostly lifted from the documentation for subprocess):

p = subprocess.Popen(my_cmd, shell=True)
os.waitpid(p.pid, 0)

OTOH, you can avoid system calls entirely:

import shutil

with open('myfile', 'w') as outfile:
    for infile in ('file1', 'file2', 'file3'):
        shutil.copyfileobj(open(infile), outfile)
7
  • 1
    It works, but let me ask you then: What's the point of the subprocess library if os.system already gets the job done? I get the feeling I should've been using subprocess instead since it's a library dedicated to this task, although since I'm doing this just for myself I'll be fine using os.system this time. Commented Feb 11, 2011 at 2:53
  • The subprocess library is much more flexible than os.system, and can model os.system precisely, but it is also more complex to work with. Commented Feb 11, 2011 at 2:56
  • 13
    os.system came before subprocess. The former is a legacy API that the latter intends to replace.
    – Santa
    Commented Feb 11, 2011 at 3:27
  • 5
    @catatemypythoncode: you should not use os.system() or shell=True. To redirect output of a subprocess, use stdout parameter as shown in Ryan Thompson's answer. Though you don't need a subprocess (cat) in your case, you could concatenate files using pure Python.
    – jfs
    Commented May 25, 2015 at 7:05
  • 5
    OTOH = On the other hand
    – Cephlin
    Commented Mar 9, 2016 at 17:49
6

@PoltoS I want to join some files and then process the resulting file. I thought using cat was the easiest alternative. Is there a better/pythonic way to do it?

Of course:

with open('myfile', 'w') as outfile:
    for infilename in ['file1', 'file2', 'file3']:
        with open(infilename) as infile:
            outfile.write(infile.read())
2
size = 'ffprobe -v error -show_entries format=size -of default=noprint_wrappers=1:nokey=1 dump.mp4 > file'
proc = subprocess.Popen(shlex.split(size), shell=True)
time.sleep(1)
proc.terminate() #proc.kill() modify it by a suggestion
size = ""
with open('file', 'r') as infile:
    for line in infile.readlines():
        size += line.strip()

print(size)
os.remove('file')

When you use subprocess , the process must be killed.This is an example.If you don't kill the process , file will be empty and you can read nothing.It can run on Windows.I can`t make sure that it can run on Unix.

3
  • 1
    It is a bad code example (it won't work on Unix; it demonstrates bad practices for line in .readlines():, s +=) and proc.kill() may lead to information loss in general (it doesn't allow the subprocess to terminate gracefully (on Unix) -- unflushed content is lost). Anyway, the note about buffering is more appropriate as a comment.
    – jfs
    Commented Oct 9, 2015 at 9:07
  • I run it on Windows is OK(Because kill is equal to terminate on Windows). On Unix may be you should use proc.terminate(). @ J.F. Sebastian I haven`t Unix System on my computer.
    – wyx
    Commented Oct 10, 2015 at 9:58
  • If you are on Windows then drop shlex.split(), drop shell=True, drop >file, drop open(), etc and use stdout=PIPE, Timer(1, proc.terminate).start(); output = proc.communicate()[0] instead. Here's complete example. More solutions: Stop reading process output in Python without hang? Note: there is no requirement in the question that you need to terminate the child process manually -- you could address other issues e.g., a process may behave differently if its stdout is a tty but it is off-topic.
    – jfs
    Commented Oct 10, 2015 at 10:04
2

It will work if your args will look like ['sh', '-c', 'cat file1 file2 file3 > myfile'] it will mean that output of cat won't pass Python and spawn in shell instead (insted of sh -c you can use bash -c)

1

One interesting case would be to update a file by appending similar file to it. Then one would not have to create a new file in the process. It is particularly useful in the case where a large file need to be appended. Here is one possibility using teminal command line directly from python.

import subprocess32 as sub

with open("A.csv","a") as f:
    f.flush()
    sub.Popen(["cat","temp.csv"],stdout=f)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.