I'm trying to port a shell script to the much more readable python version. The original shell script starts several processes (utilities, monitors, etc.) in the background with "&". How can I achieve the same effect in python? I'd like these processes not to die when the python scripts complete. I am sure it's related to the concept of a daemon somehow, but I couldn't find how to do this easily.
9 Answers
While jkp's solution works, the newer way of doing things (and the way the documentation recommends) is to use the subprocess
module. For simple commands its equivalent, but it offers more options if you want to do something complicated.
Example for your case:
import subprocess
subprocess.Popen(["rm","-r","some.file"])
This will run rm -r some.file
in the background. Note that calling .communicate()
on the object returned from Popen
will block until it completes, so don't do that if you want it to run in the background:
import subprocess
ls_output=subprocess.Popen(["sleep", "30"])
ls_output.communicate() # Will block for 30 seconds
See the documentation here.
Also, a point of clarification: "Background" as you use it here is purely a shell concept; technically, what you mean is that you want to spawn a process without blocking while you wait for it to complete. However, I've used "background" here to refer to shell-background-like behavior.
-
14@Dan: How do I kill the process once it's running in the background? I want to run it for a while (it's a daemon that I interact with) and kill it when I'm done with it. The docs aren't helpful...– JuanJul 7, 2014 at 5:36
-
1@Dan but don't I need to know the PID for that? Activity monitor/Task manager not an option (needs to happen programmatically).– JuanJul 12, 2014 at 16:56
-
5ok so how do you force the process to background when you need the result of Popen() to write to its stdin?– MichaelJul 19, 2014 at 20:23
-
2@J.F.Sebastian: I interpreted it as "how can I create an independent process that doesn't stop the execution of the current program". How would you suggest I edit it to make this more explicit?– DanDec 17, 2014 at 4:51
-
8@Dan
proc = subprocess.Popen(["rm","-r","some.file"])
, then to kill:proc.terminate()
– A TJun 9, 2015 at 4:52
Note: This answer is less current than it was when posted in 2009. Using the subprocess
module shown in other answers is now recommended in the docs
(Note that the subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using these functions.)
If you want your process to start in the background you can either use system()
and call it in the same way your shell script did, or you can spawn
it:
import os
os.spawnl(os.P_DETACH, 'some_long_running_command')
(or, alternatively, you may try the less portable os.P_NOWAIT
flag).
See the documentation here.
-
9Remark: you must specify the full path to the executable. This function will not use the PATH variable and the variant that does use it is not available under Windows.– sorinOct 28, 2009 at 17:14
-
39
-
33
-
14Could the people suggesting using
subprocess
give us a hint how to detach a process withsubprocess
?– raksliceFeb 11, 2014 at 21:56 -
3How can I use Python script (say attach.py) to find a background process and redirect its IO so that attach.py can read from / write to some_long_running_prog in background?– raof01Mar 5, 2014 at 6:46
You probably want the answer to "How to call an external command in Python".
The simplest approach is to use the os.system
function, e.g.:
import os
os.system("some_command &")
Basically, whatever you pass to the system
function will be executed the same as if you'd passed it to the shell in a script.
-
11IMHO, python scripts are usually written to be cross-platform and if there simple cross-platform solution exists it's better to stick with it. Never know if you'll have to work with another platform in future :) Or if some other man would want to migrate your script to his platform.– d9kMay 1, 2016 at 2:37
-
8This command is synchronous (i.e. it always waits for a termination of the started process).– tavOct 7, 2017 at 9:10
-
-
1@d9k isn't the choice of running something in the background already positioning you in posix-land? What would you do on Windows? Run as a service? Jun 4, 2018 at 17:57
-
how can I use this if I need to run a command from a specific folder?– mrRobotNov 16, 2019 at 16:14
I found this here:
On windows (win xp), the parent process will not finish until the longtask.py
has finished its work. It is not what you want in CGI-script. The problem is not specific to Python, in PHP community the problems are the same.
The solution is to pass DETACHED_PROCESS
Process Creation Flag to the underlying CreateProcess
function in win API. If you happen to have installed pywin32 you can import the flag from the win32process module, otherwise you should define it yourself:
DETACHED_PROCESS = 0x00000008
pid = subprocess.Popen([sys.executable, "longtask.py"],
creationflags=DETACHED_PROCESS).pid
-
8+1 for showing how to retain the process id. And if anyone want to kill the program later with the process id: stackoverflow.com/questions/17856928/…– iChuxSep 17, 2015 at 10:38
-
5
Use subprocess.Popen()
with the close_fds=True
parameter, which will allow the spawned subprocess to be detached from the Python process itself and continue running even after Python exits.
https://gist.github.com/yinjimmy/d6ad0742d03d54518e9f
import os, time, sys, subprocess
if len(sys.argv) == 2:
time.sleep(5)
print 'track end'
if sys.platform == 'darwin':
subprocess.Popen(['say', 'hello'])
else:
print 'main begin'
subprocess.Popen(['python', os.path.realpath(__file__), '0'], close_fds=True)
print 'main end'
-
1In windows, it doesn't detach but using creationflags parameter works Apr 26, 2018 at 15:43
-
4
-
@TitanFighter this can be avoid by set SIGCHLD SIG_IGN : stackoverflow.com/questions/16807603/… May 30, 2020 at 5:20
-
1
-
The
close_fds=True
option works by detaching the process, but it didn't return back to my Python program. Hoping to find an option that truly executes a process and sends it to the background and then returns back to the Python program.– NavApr 27, 2021 at 7:27
Both capture output and run on background with threading
As mentioned on this answer, if you capture the output with stdout=
and then try to read()
, then the process blocks.
However, there are cases where you need this. For example, I wanted to launch two processes that talk over a port between them, and save their stdout to a log file and stdout.
The threading
module allows us to do that.
First, have a look at how to do the output redirection part alone in this question: Python Popen: Write to stdout AND log file simultaneously
Then:
main.py
#!/usr/bin/env python3
import os
import subprocess
import sys
import threading
def output_reader(proc, file):
while True:
byte = proc.stdout.read(1)
if byte:
sys.stdout.buffer.write(byte)
sys.stdout.flush()
file.buffer.write(byte)
else:
break
with subprocess.Popen(['./sleep.py', '0'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc1, \
subprocess.Popen(['./sleep.py', '10'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc2, \
open('log1.log', 'w') as file1, \
open('log2.log', 'w') as file2:
t1 = threading.Thread(target=output_reader, args=(proc1, file1))
t2 = threading.Thread(target=output_reader, args=(proc2, file2))
t1.start()
t2.start()
t1.join()
t2.join()
sleep.py
#!/usr/bin/env python3
import sys
import time
for i in range(4):
print(i + int(sys.argv[1]))
sys.stdout.flush()
time.sleep(0.5)
After running:
./main.py
stdout get updated every 0.5 seconds for every two lines to contain:
0
10
1
11
2
12
3
13
and each log file contains the respective log for a given process.
Inspired by: https://eli.thegreenplace.net/2017/interacting-with-a-long-running-child-process-in-python/
Tested on Ubuntu 18.04, Python 3.6.7.
You probably want to start investigating the os module for forking different threads (by opening an interactive session and issuing help(os)). The relevant functions are fork and any of the exec ones. To give you an idea on how to start, put something like this in a function that performs the fork (the function needs to take a list or tuple 'args' as an argument that contains the program's name and its parameters; you may also want to define stdin, out and err for the new thread):
try:
pid = os.fork()
except OSError, e:
## some debug output
sys.exit(1)
if pid == 0:
## eventually use os.putenv(..) to set environment variables
## os.execv strips of args[0] for the arguments
os.execv(args[0], args)
-
2
os.fork()
is really useful, but it does have a notable downside of only being available on *nix. Jul 28, 2009 at 19:15 -
-
More details about this approach: Creating a daemon the Python way Apr 20, 2014 at 2:47
-
1You can also reach similar effects with
threading
: stackoverflow.com/a/53751896/895245 I think that might work on Windows. Dec 12, 2018 at 21:53
You can use
import os
pid = os.fork()
if pid == 0:
Continue to other code ...
This will make the python process run in background.
I haven't tried this yet but using .pyw files instead of .py files should help. pyw files dosen't have a console so in theory it should not appear and work like a background process.
-
This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review Feb 15 at 12:12
subprocess.Popen()
is the new recommended way since 2010 (we are in 2015 now) and (3) the duplicated question redirecting here has also an accepted answer aboutsubprocess.Popen()
. Cheers :-)subprocess.Popen("<command>")
with <command> file led by a suitable shebang. Works perfect for me (Debian) with bash and python scripts, implicitelyshell
s and survives its parent process.stdout
goes to same terminal than the parent's. So this works much like&
in a shell which was OPs request. But hell, all the questions work out very complex while a little testing showed it in no time ;)