5

Django only allows you to use one database in settings.py. Does that prevent you from scaling up? (millions of users)

5
  • If you're really concerned about maximum performance, you don't want to be using a framework at all.
    – JAL
    Jan 24, 2010 at 13:39
  • 2
    @Code Duck - Note that there is a difference between scalability and maximum performance. Most of the time, frameworks probably aid scalability more than they hurt it because they allow you to focus on higher-level issues. And there are a number of large enterprises that scale using django. Jan 24, 2010 at 13:44
  • There's no doubt it aids development. However, I've heard of plenty of sites that had to rewrite for performance once their traffic really took off. In my tests on a simple site, using the same DB, straight PHP with no cache can serve 3 times as many req/sec as Django with memached.
    – JAL
    Jan 24, 2010 at 16:34
  • @Code Duck - Your comment doesn't make a whole lot of sense. If we were all concerned about "maximum performance", we'd be writing our web apps in assembler. There's sane level of trade-off between developer efficiency and raw performance though.
    – jamieb
    Jan 24, 2010 at 19:36
  • Why yes, there's a tradeoff.... regardless, sometimes sites have to rewrite portions without a framework or even in C for efficiency, regardless of the drawbacks, due to the extra overhead that comes from using a standard framework. I'm thinking of one in particular that started off in Django but had to redo most everything once they were at about 200m pages a month. Of course, 99.5% of sites never have this problem. You can always work on the DB layer or just throw more hardware at the problem, I guess.
    – JAL
    Jan 24, 2010 at 22:09

6 Answers 6

11

Django now has support for multiple databases.

7

The database isn't your bottleneck.

Check your browser carefully.

For each page of HTML you're sending (on average) 8 other files, some of which may be quite large. These are your JS, CSS, graphics, etc.

The actual performance bottleneck is the browser requesting those files and accepting the bytes s... l... o... w... l... y...

To scale, then, do this.

  1. Use multiple front-ends balanced with a pure software solution like wackamole. http://www.backhand.org/wackamole/

  2. Use proxy servers like squid to send the "other" files. They're largely static. This is where 7/8ths of the work is done downloading to the client. Don't scrimp on getting these right.

  3. Use multiple, concurrent mod_wsgi/Django to create the -- rare -- piece of dynamic HTML based on DB queries. Be sure that mod_wsgi is in daemon mode so that you can have multiple Django servers available to Apache. Build as many of these as you need. They're all identical, all in parallel, and all shared by Wackamole.

  4. Use a single, fast database like MySQL for the few things that must come from a database. MySQL will make use of multiple cores on it's server, so it will scale reasonably well without you having to do anything other than buy memory. Put this on a separate box, all by itself, dedicated and tuned for just this.

You'll find that this scales nicely. You'll find that the load is shared nicely between squid, apache, the Django daemons and the actual database. You'll also find that each part of the load (from the boring static parts to the interesting database query) happens separately and concurrently.

Finally, buy Schlossnagle's book. http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X

5
  • This is good advice, but it seems to be somewhat situational. At a certain point, having a single database is a recipe for a scalability problem. Granted, how easy that point is to reach is dependent upon the situation. Jan 24, 2010 at 13:36
  • @Jason Baker: I can't see why a single database has to be a limitation. With commercial products like Oracle and DB2, you can have a single database that spans multiple processors (each with multiple cores). Why is single database a limitation?
    – S.Lott
    Jan 24, 2010 at 13:40
  • @S.Lott - Single anything is a limitation in terms of scalability. First of all, with a single database you have a single point of failure. Secondly, it's not just CPU time that's limiting to a database. There are also I/O issues to deal with. That said, it's very much possible (even probable) that you don't need to scale to the point where that becomes an issue. But it does become a problem at a certain point. Jan 24, 2010 at 13:53
  • @Jason Baker: I'm pretty sure that when you say "single" database, you're not looking at the clustered solutions available from the various sources. mysql.com/products/database/cluster It appears as a "single" database to your application. I'm not sure what you're talking about because you've moved from scalability to reliability. But I think you're ignoring the offerings that address this.
    – S.Lott
    Jan 24, 2010 at 14:50
  • Database has always been what's limited me. I'm running on 20+ nginx webservers and about 5 postgres servers. Currently experimenting with hbase and oracle. I'll be checking into mysqlcluster as well, as S.Lott mentioned. Too bad postgres doesn't have something similar.
    – Gattster
    Jun 18, 2013 at 22:13
3

Read scaling to millions of users is not a database problem, but is fixed with load balancing and caching, etc, see S. Lott above.

Write scaling can indeed be a database problem. "Sharding" and having multiple databases can be one solution, but that's hard with SQL while still retaining the relationality of the database. Popular solutions there are the new types of "nosql" databases. But if you really have those problems, then you need serious expert help, not just answers from dudes Stackoverflow. :)

1
  • I've been trying nosql solutions for a while. One of my projects is at the point where we're rewriting old pieces of it into hbase / redis solutions to free up our database from too many writes. Yes, it's a good problem to have, but this is not a very fun process!
    – Gattster
    Jun 18, 2013 at 22:17
1

Some great answers already (S. Lott for example), however I thought I should pipe in with some more things:

Make sure not to use the database for logical operations

I understand the attractiveness of Order By or SQL Procedures however you only have one database but you have multiple django servers, let the servers handle this if you can.

Of course, if you only want the last ten rows according to a certain criterion (date), then by all means do precise it in the request ;) Just make sure not to overload your database with operations that could be handled elsewhere.

Throw more hardware to the problem

MySQL and Oracle scale quite well with hardware, if you have a small problem of performance you could begin by adding more hardware.

Split your database

I know that for relationships and all you have to manage some tables together, however if you ever have a load problem, try to group your tables, for example if you have a "history" group of tables, perhaps that it could work without the others and be on a separate server.

Do consider tuning, and watch out for your requests/index

You would need experts advises here, but I can tell from experience that even a single badly tuned request can wreak havoc... and it's quite difficult to find out. You can consider the Ask Tom website for example of diagnosis / fine tuning.

Don't decide on your tables architecture in isolation, but do consider the requests

Hierarchical requests and multiple joins can be really costly. You don't have to build a fully normalized relations schema and may consider some denormalization in order to better accomodate the type of requests the database will face.

Just a couple of thoughts :)

1

A few miscellaneous pieces of advice:

  • I'm surprised no one's mentioned this yet. Use memcached. If you're getting a lot of repetitive types of queries (which most webapps do), this can make a huge difference.

  • Consider using Oracle's failover and load balancing. It allows you to add support for multiple databases on a single db connection.

  • Another thing to consider is using a system similar to FriendFeed's. This solves the problem of "how do we make changes to the database without halting the world?" more than anything else.

0

If you find out that the DB is the bottlenck of your app, and their is now way around it (like using caching) then you should scale your DB as well. Django has nothing to do with this

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.