3

I am trying to write an application that will allow a user to launch a fairly long-running process (5-30 seconds). It should then allow the user to check the output of the process as it is generated. The output will only be needed for the user's current session so nothing needs to be stored long-term. I have two questions regarding how to accomplish this while taking advantage of the Pylons framework:

  1. What is the best way to launch a background process such as this with a Pylons controller?

  2. What is the best way to get the output of the background process back to the user? (Should I store the output in a database, in session data, etc.?)

Edit: The problem is if I launch a command using subprocess in a controller, the controller waits for the subprocess to finish before continuing, showing the user a blank page that is just loading until the process is complete. I want to be able to redirect the user to a status page immediately after starting the subprocess, allowing it to complete on its own.

2 Answers 2

6

I've handled this problem in the past (long-running process invoked over HTTP) by having my invoked 2nd process daemonize. Your Pylons controller makes a system call to your 2nd process (passing whatever data is needed) and the 2nd process immediately becomes a daemon. This ends the system call and your controller can return.

My web-apps usually then issue AJAX requests to "check-on" the daemon process until it has completed. I've used both tmp files (cPickle works well) and databases to share information between the daemon and the web-app.

Excellent python daemon recipe: http://code.activestate.com/recipes/278731/

2
  • I am hitting this problem. Is ths safe? What if you have 10000 users every minute? And most of the requests are very expensive? Then would you have many daemon running? Also, are you suggesting running a daemon pylon process? ThAnks.
    – User007
    Jul 22, 2012 at 12:42
  • Is this safe? Yes, I've been using this in a production environment for years. Just make sure your daemon processes exit cleanly. My load is probably only a couple dozen users per hour, though. At 10,000 users per hour, is any method to a launch long-running process going to be safe? Can your server even handle that sort of memory/CPU usage? If you didn't fork out a 2nd process, could your web-server setup even handle it?
    – Mark
    Jul 22, 2012 at 15:27
1

I think this has little to do with pylons. I would do it (in whatever framework) in these steps:

  • generate some ID for the new job, and add a record in the database.
  • create a new process, e.g. through the subprocess module, and pass the ID on the command line (*).
  • have the process write its output to /tmp/project/ID
  • in pylons, implement URLs of the form /job/ID or /job?id=ID. That will look into the database whether the job is completed or not, and merge the temporary output into the page.

(*) It might be better for the subprocess to create another process immediately, and have the pylons process wait for the first child, so that there will be no zombie processes.

1
  • Predictable temporary file names are often unsafe -- they can be abused to overwrite arbitrary files to which your web server has write access. I would suggest using a pre-created (with appropriate permissions) directory under /var, say. Sep 17, 2009 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.