7

In one model I've got update() method which updating few fields and creates one object of some other model. The problem is that data I use to update is fetched from another host (unique for each object) and it could take a moment (host may be offline, and timeout is set to 3sec). And now, I need to update couple of hundred objects, 3-4 times per hour - of course updating every one in a row is not an option, because it could take all day. My first thought was split it up for 50-100 threads so each one could update its own part of objects. 99% of update function time is waiting for server respond (there is few bytes of data only, so pings are the problem), I think the CPU won't be a problem, I'm more worried about:

  • Django ORM. Can it handle it? Getting all objects, splitting it up, and updating from >50 threads?
  • Is it a good idea to solve this? If it is - how to do it and don't screw a database? Or maybe I shouldn't care about so little records?
  • If it isn't a good way, how to do it right?
1
  • spawning 50-100 threads isn't going to help much... If you're in fact doing threading in python, you will hit the GIL and each thread will be stepping on eachother (aka... it won't be handled in parallel). Even if you're using multiprocessing to handle this issue, you're going to have the SAME exact issue just at the OS level with the OS context switching between processes as I doubt your system has 50 cpu cores...
    – g19fanatic
    Aug 24, 2012 at 16:29

2 Answers 2

8

You can perform actions from different thread manually (eg with Queue and executors pool), but you should note, that Django's ORM manages database connections in thread-local variables. So each new thread = new connection to database (which will be not good idea for 50-100 threads for one request - too many connections). On the other hand, you should check database "bandwith".

7
  • So, Django's ORM isn't thread-safe? I can't get objects of some model ,pass them to different threads and do some operations on them, yes?
    – Kiro
    Aug 24, 2012 at 23:25
  • 1
    Passing resources between threads is generally a bad idea, instead you should be passing parameters and let the thread fetch its own objects.
    – thebjorn
    Aug 25, 2012 at 8:32
  • @thebjorn I see, but now, what with too many connection problem? 100 connections in one time? It's also bad idea, isn't it? So, how should I do it?
    – Kiro
    Aug 25, 2012 at 13:08
  • 1
    @Kiro whether 100 connections at one time is too much depends on many factors (which database you're using, what your load is, whether you can control when this happens, etc.). The threading.Queue class and using the producer/consumer pattern will let you control the number of simultaneous connections if that is a concern.
    – thebjorn
    Aug 25, 2012 at 20:27
  • @thebjorn There are no threading.Queue in python, I think you mean Queue.Queue one. Aug 25, 2012 at 20:32
2

Threads should work wonderfully for this kind of work. (@g19fanatic: the GIL is not going to be a problem, of course, since these tasks are not cpu-bound -- it's the same reason that there is no point in using multiprocessing.. or worrying about the number of cores)

The Django ORM can handle this, but depending on what you're doing you might need to use transactions -- but try not to hold a transaction open for 3 seconds if you can avoid it.

Normally, I would suggest using threading.Queue and a producer/consumer pattern (e.g. the bottom of this page), however, since you know that the number of tasks is reasonable and your tasks take a long time (3 seconds) you might as well just spawn them all and let the OS figure it out :-)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.