You did not specify many requirements and constraints. I'm going to assume you already have a redis instance somewhere.
What you can do is read the output from the other process line by line and publish it through redis:
Here's an example where you can echo
data into a file /tmp/foo
for testing:
import redis
redis_instance = redis.Redis()
p = subprocess.Popen(shlex.split("tail -f /tmp/foo"), stdout=subprocess.PIPE)
while True:
line = p.stdout.readline()
if line:
redis_instance.publish('process log', line)
else:
break
In a separate process:
import redis
redis_instance = redis.Redis()
pubsub = redis_instance.pubsub()
pubsub.subscribe('process log')
while True:
for message in pubsub.listen():
print message # or use websockets to comunicate with a browser
If you want the process to end, you can e.g. send a "quit" after the celery task is done.
You can use different channels (the string in subscribe
) to separate the output from different processes.
You can also store your log output in redis, if you want to,
redis_instance.rpush('process log', message)
and later retrieve it in full.
subprocess.call
?