4

Newb quesion about Django app design:

Im building reporting engine for my web-site. And I have a big (and getting bigger with time) amounts of data, and some algorithm which must be applied to it. Calculations promise to be heavy on resources, and it would be stupid if they are performed by requests of users. So, I think to put them into background process, which would be executed continuously and from time to time return results, which could be feed to Django views-routine for producing html output by demand.

And my question is - what proper design approach for building such system? Any thoughts?

1
  • 1
    Did you look into Celery? It can be used to schedule tasks! Dec 9, 2012 at 0:48

3 Answers 3

3

Celery is one of your best choices. We are using it successfully. It has a powerful scheduling mechanism - you can either schedule tasks as a timed job or trigger tasks in background when user (for example) requests it.

It also provides ways to query for the status of such background tasks and has a number of flow control features. It allows for a very easy distribution of the work - i.e your celery background tasks can be run on a separate machine (this is very useful for example with heroku web/workers split where web process is limited to max 30s per request). It provides various queue backends (it can use database, rabbitMQ or a number of other queuing mechanisms. With simplest setup it can use the same database that your Django site already uses for that (which makes it easy to setup).

And if you are using automated tests it also has a feature that helps with testing - it can be set in "eager" mode, where background tasks are not executed in background - thus giving predictable logic testing.

More info here: http://docs.celeryproject.org:8000/en/latest/django/

0

You mean the results are returned into a database or do you want to create django-views directly from your independently running code?

If you have large amounts of data I like to use Pythons multiprocessing. You can create a Generator which fills a JoinableQueue with the different tasks to do and a pool of Workers consuming the different Tasks. This way you should be able to maximize the resource utilization on your system.

The multiprocessing module also allows you to do several tasks over the network (e.g. multiprocessing.Manager()). With this in mind you should easily be able to scale things up if you need a second machine to process the data in time.

Example:

This example shows how to spawn multiple processes. The generator function should query the database for all new entries that need heavy lifting. The consumers take the individual items from the queue and do the actual calculations.

import time 

from multiprocessing.queues import JoinableQueue
from multiprocessing import Process

QUEUE = JoinableQueue(-1)

def generator():
    """ Puts items in the queue. For example query database for all new, 
    unprocessed entries that need some serious math done.."""
    while True: 
        QUEUE.put("Item")
        time.sleep(0.1)


def consumer(consumer_id):
    """ Consumes items from the queue... Do your calculations here... """
    while True: 
        item = QUEUE.get()
        print "Process %s has done: %s" % (consumer_id, item)
        QUEUE.task_done()


p = Process(target=generator)
p.start()

for x in range(0, 2): 
    w = Process(target=consumer, args=(x,))
    w.start()

p.join()
w.join()
6
  • "You mean the results are returned into a database or do you want to create django-views directly from your independently running code?" Than seamlessly they will be integrated than better.
    – Gill Bates
    Dec 8, 2012 at 20:13
  • Since you said that the amount of data that needs to be processed is big I would suggest you save the data into a database and query the database from django. This way you don't need to create an extra way of communication if you use multiprocessing and the data is persisted. You can only use parts of django to make the db access from the other code-parts as discussed in stackoverflow.com/questions/937742/use-django-orm-as-standalone and stackoverflow.com/questions/302651/… for example.
    – chkorn
    Dec 8, 2012 at 20:41
  • 1
    My confusion is in how that background process can be called within django, takin in attention that it must be unbounded from request - how this process can be started in server startup (or by schedule), and executed continously?
    – Gill Bates
    Dec 8, 2012 at 20:54
  • Ah! Now I get it. Sorry for the confusion. I suggest you just set a flag in your database that says "Work needs to be done". The background task(s) then just queries your database for the list of open tasks. I have added an example with a consumer/generator that should demonstrate the principle a little bit. This proccess would be started by hand or as a Service/Daemon. Making it run continuously can be done by an infinite loop (e.g. while True:). Just make sure you have some kind of stop-condition so that you can terminate gracefully :)
    – chkorn
    Dec 8, 2012 at 21:26
  • Can this process be putted in Django runtime, or its bad decision? ps Thanks you for example by the way!
    – Gill Bates
    Dec 8, 2012 at 21:36
0

Why don't you have a url or python script that triggers whatever sort of calculation you need to have done everytime it's run and then fetch that url or run that script via a cronjob on the server? From what your question was it doesn't seem like you need a whole lot more than that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.