61

I'm using django with apache and mod_wsgi and PostgreSQL (all on same host), and I need to handle a lot of simple dynamic page requests (hundreds per second). I faced with problem that the bottleneck is that a django don't have persistent database connection and reconnects on each requests (that takes near 5ms). While doing a benchmark I got that with persistent connection I can handle near 500 r/s while without I get only 50 r/s.

Anyone have any advice? How can I modify Django to use a persistent connection or speed up the connection from Python to DB?

7 Answers 7

37

Django 1.6 has added persistent connections support (link to doc for latest stable Django ):

Persistent connections avoid the overhead of re-establishing a connection to the database in each request. They’re controlled by the CONN_MAX_AGE parameter which defines the maximum lifetime of a connection. It can be set independently for each database.

1
27

Try PgBouncer - a lightweight connection pooler for PostgreSQL. Features:

  • Several levels of brutality when rotating connections:
    • Session pooling
    • Transaction pooling
    • Statement pooling
  • Low memory requirements (2k per connection by default).
3
  • 1
    Same as pgpool it won't eliminate each request connect overhead which is huge. The connection opening code is a real bottleneck. Nov 9, 2009 at 12:48
  • 4
    @Mike TK First, it is not the same as pgpool. Pgbouncer works with libevent and manages connections asynchroniously as opposed to pgpool forking a process for each connection as Postgres itself does (the only difference is pgpool keeps the processes alive). From my experience using pgbouncer (compared to not using any method at all) gives a noticeable speedup.
    – Vasil
    Jun 7, 2010 at 23:59
  • 5
    +1. This is what they use at Disqus (to date, the Django project with the largest user base). I'd regard that as fire tested.
    – tawmas
    Jul 15, 2011 at 19:35
20

In Django trunk, edit django/db/__init__.py and comment out the line:

signals.request_finished.connect(close_connection)

This signal handler causes it to disconnect from the database after every request. I don't know what all of the side-effects of doing this will be, but it doesn't make any sense to start a new connection after every request; it destroys performance, as you've noticed.

I'm using this now, but I havn't done a full set of tests to see if anything breaks.

I don't know why everyone thinks this needs a new backend or a special connection pooler or other complex solutions. This seems very simple, though I don't doubt there are some obscure gotchas that made them do this in the first place--which should be dealt with more sensibly; 5ms overhead for every request is quite a lot for a high-performance service, as you've noticed. (It takes me 150ms--I havn't figured out why yet.)

Edit: another necessary change is in django/middleware/transaction.py; remove the two transaction.is_dirty() tests and always call commit() or rollback(). Otherwise, it won't commit a transaction if it only read from the database, which will leave locks open that should be closed.

3
  • 5
    See groups.google.com/group/django-users/browse_thread/thread/… django-postgres-persistent-db-connection.diff for a more general implementation of this, but it's only implemented for Postgresql. (Not that the OP is even listening, but for anyone else that finds their way here...) Jul 29, 2009 at 1:51
  • 1
    One of the reasons people suggest new backends is code portability; every upgrade means you have to edit the code again or keep your branch in sync if you start commenting out the django codebase. Dec 5, 2011 at 6:17
  • 1
    I check the source code in django 2.0 and found that django only close the connections which are past their lifetime.Things are different maybe.Your example code is no longer exists and new code is "signals.request_finished.connect(close_old_connections) "
    – dogewang
    Dec 6, 2018 at 5:52
16

I created a small Django patch that implements connection pooling of MySQL and PostgreSQL via sqlalchemy pooling.

This works perfectly on production of http://grandcapital.net/ for a long period of time.

The patch was written after googling the topic a bit.

1
  • +1 Many thanks Igor, I used your patch and it worked flawlessly. I found another reference to the same approach at menendez.com/blog/… but yours was much simpler to put to work. Oct 20, 2011 at 19:04
4

Disclaimer: I have not tried this.

I believe you need to implement a custom database back end. There are a few examples on the web that shows how to implement a database back end with connection pooling.

Using a connection pool would probably be a good solution for you case, as the network connections are kept open when connections are returned to the pool.

  • This post accomplishes this by patching Django (one of the comments points out that it is better to implement a custom back end outside of the core django code)
  • This post is an implementation of a custom db back end

Both posts use MySQL - perhaps you are able to use similar techniques with Postgresql.

Edit:

  • The Django Book mentions Postgresql connection pooling, using pgpool (tutorial).
  • Someone posted a patch for the psycopg2 backend that implements connection pooling. I suggest creating a copy of the existing back end in your own project and patching that one.
1
  • I've tried pgpool, but this not improved situation a lot (I still need reconnect each time). PgPool is designed for a bit other purposes (failover, replication etc). Jul 14, 2009 at 14:14
1

This is a package for django connection pool: django-db-connection-pool

pip install django-db-connection-pool

You can provide additional options to pass to SQLAlchemy's pool creation, key's name is POOL_OPTIONS:

DATABASES = {
    'default': {
        ...
        'POOL_OPTIONS' : {
            'POOL_SIZE': 10,
            'MAX_OVERFLOW': 10
        }
        ...
     }
 }
0

I made some small custom psycopg2 backend that implements persistent connection using global variable. With this I was able to improve the amout of requests per second from 350 to 1600 (on very simple page with few selects) Just save it in the file called base.py in any directory (e.g. postgresql_psycopg2_persistent) and set in settings

DATABASE_ENGINE to projectname.postgresql_psycopg2_persistent

NOTE!!! the code is not threadsafe - you can't use it with python threads because of unexpectable results, in case of mod_wsgi please use prefork daemon mode with threads=1


# Custom DB backend postgresql_psycopg2 based
# implements persistent database connection using global variable

from django.db.backends.postgresql_psycopg2.base import DatabaseError, DatabaseWrapper as BaseDatabaseWrapper, \
    IntegrityError
from psycopg2 import OperationalError

connection = None

class DatabaseWrapper(BaseDatabaseWrapper):
    def _cursor(self, *args, **kwargs):
        global connection
        if connection is not None and self.connection is None:
            try: # Check if connection is alive
                connection.cursor().execute('SELECT 1')
            except OperationalError: # The connection is not working, need reconnect
                connection = None
            else:
                self.connection = connection
        cursor = super(DatabaseWrapper, self)._cursor(*args, **kwargs)
        if connection is None and self.connection is not None:
            connection = self.connection
        return cursor

    def close(self):
        if self.connection is not None:
            self.connection.commit()
            self.connection = None

Or here is a thread safe one, but python threads don't use multiple cores, so you won't get such performance boost as with previous one. You can use this one with multi process one too.

# Custom DB backend postgresql_psycopg2 based
# implements persistent database connection using thread local storage
from threading import local

from django.db.backends.postgresql_psycopg2.base import DatabaseError, \
    DatabaseWrapper as BaseDatabaseWrapper, IntegrityError
from psycopg2 import OperationalError

threadlocal = local()

class DatabaseWrapper(BaseDatabaseWrapper):
    def _cursor(self, *args, **kwargs):
        if hasattr(threadlocal, 'connection') and threadlocal.connection is \
            not None and self.connection is None:
            try: # Check if connection is alive
                threadlocal.connection.cursor().execute('SELECT 1')
            except OperationalError: # The connection is not working, need reconnect
                threadlocal.connection = None
            else:
                self.connection = threadlocal.connection
        cursor = super(DatabaseWrapper, self)._cursor(*args, **kwargs)
        if (not hasattr(threadlocal, 'connection') or threadlocal.connection \
             is None) and self.connection is not None:
            threadlocal.connection = self.connection
        return cursor

    def close(self):
        if self.connection is not None:
            self.connection.commit()
            self.connection = None
3
  • 3
    Please don't do this. This is completely unthreadsafe. Use proper connection pooling like pgpool. Aug 29, 2009 at 15:04
  • Pgpool won't help because django need to reconnect every time anyway. I know that it's not threadsafe code (and I have a threadsafe version that uses psycopg2.pool module, just not published yet), but I use python with mod_wsgi and in Daemon mode witout threads but with pure prefork, so it's safe here. I'll add the note - thanks. Aug 31, 2009 at 10:39
  • use pgbouncer locally, then whenever django connects, pgbouncer will use existing connection from the pool. as you will have localhost connection to the pgbouncer, there will be no big penalty on connection
    – Jerzyk
    Apr 2, 2011 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.