81

I have a python script that has to launch a shell command for every file in a dir:

import os

files = os.listdir(".")
for f in files:
    os.execlp("myscript", "myscript", f)

This works fine for the first file, but after the "myscript" command has ended, the execution stops and does not come back to the python script.

How can I do this? Do I have to fork() before calling os.execlp()?

1

7 Answers 7

127

subprocess: The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.

http://docs.python.org/library/subprocess.html

Usage:

import subprocess
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
process.wait()
print process.returncode
4
  • 1
    I think as the accepted answer, this should contain at least the same amount of detail as @Harley. This is more of a personal request, but I think the newer docs present the information better. Could you link to the 2.7 version of the documentation instead? Aug 3, 2011 at 20:58
  • 3
    The newer version of the documentation is located at docs.python.org/library/subprocess.html .
    – wanderso
    Sep 29, 2011 at 17:29
  • 4
    remove stdout=PIPE it might hang the script if the child program produces enough output (around 65KB on Linux). shell=True is unnecessary.
    – jfs
    Nov 22, 2012 at 16:11
  • 2
    Without shell=True it raises an exception on python 2.7... File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory
    – SeF
    Jun 7, 2017 at 14:54
66

You can use subprocess.Popen. There's a few ways to do it:

import subprocess
cmd = ['/run/myscript', '--arg', 'value']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
for line in p.stdout:
    print line
p.wait()
print p.returncode

Or, if you don't care what the external program actually does:

cmd = ['/run/myscript', '--arg', 'value']
subprocess.Popen(cmd).wait()
2
  • 1
    for line in p.stdout: might delay the output of a line due to read-ahead. You could use for line in iter(p.stdout.readline, b''): print line, (note: comma at the end). Remove stdout=PIPE in the second example; it might hang the script. Or just use subprocess.call() if you don't need an exception for non-zero returned code
    – jfs
    Nov 22, 2012 at 16:17
  • @HarleyHolcombe i tried your solution via subprocess.Popen("/usr/bin/sleep 20").wait(), which didn't work (FileNotFoundError:), to make it work I had to separate the command into substrings and put them into a list subprocess.Popen(["/usr/bin/sleep", "20"]).wait() why is that necessary? Jul 23, 2022 at 10:41
13

The subprocess module has come along way since 2008. In particular check_call and check_output make simple subprocess stuff even easier. The check_* family of functions are nice it that they raise an exception if something goes wrong.

import os
import subprocess

files = os.listdir('.')
for f in files:
   subprocess.check_call( [ 'myscript', f ] )

Any output generated by myscript will display as though your process produced the output (technically myscript and your python script share the same stdout). There are a couple of ways to avoid this.

  • check_call( [ 'myscript', f ], stdout=subprocess.PIPE )
    The stdout will be supressed (beware if myscript produces more that 4k of output). stderr will still be shown unless you add the option stderr=subprocess.PIPE.
  • check_output( [ 'myscript', f ] )
    check_output returns the stdout as a string so it isnt shown. stderr is still shown unless you add the option stderr=subprocess.STDOUT.
2
  • 4
    @Luke Stanley: your question is unclear, but I'll give it a shot. The subprocess.check_* functions makes calling external binaries easier than manually using subprocess.Popen. In particular you never need to deal with wait(), communicate(), return_code, etc. You also don't need to worry about gotchas like the dangling pipe stalling the executable. When it comes to interacting with an external binary the check_* functions aren't tremendously helpful, but for most use cases the result is shorter and less error prone, aka easier.
    – deft_code
    Jun 10, 2011 at 19:03
  • 1
    you could use subprocess.DEVNULL or open(os.devnull, 'r+b') to ignore input/output instead of subprocess.PIPE to avoid possible dead-lock
    – jfs
    Nov 22, 2012 at 16:21
7

The os.exec*() functions replace the current programm with the new one. When this programm ends so does your process. You probably want os.system().

1
  • 1
    os.system() is very problematic in terms of security, to the point where many C libraries just remove this standard C function outright. subprocess.Popen() with shell=False should be used instead
    – Shnatsel
    Dec 27, 2014 at 0:28
3

use spawn

import os
os.spawnlp(os.P_WAIT, 'cp', 'cp', 'index.html', '/dev/null')
2

I use os.system

import os
os.system("pdftoppm -png {} {}".format(path2pdf, os.path.join(tmpdirname, "temp")))
1

this worked for me fine!

shell_command = "ls -l" subprocess.call(shell_command.split())

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.