73

I am trying to find out the number of queries executed by a utility function. I have written a unit test for this function and the function is working well. What I would like to do is track the number of SQL queries executed by the function so that I can see if there is any improvement after some refactoring.

def do_something_in_the_database():
    # Does something in the database
    # return result

class DoSomethingTests(django.test.TestCase):
    def test_function_returns_correct_values(self):
        self.assertEqual(n, <number of SQL queries executed>)

EDIT: I found out that there is a pending Django feature request for this. However the ticket is still open. In the meantime is there another way to go about this?

8 Answers 8

88

Since Django 1.3 there is a assertNumQueries available exactly for this purpose.

One way to use it (as of Django 3.2) is as a context manager:

# measure queries of some_func and some_func2
with self.assertNumQueries(2):
    result = some_func()
    result2 = some_func2()
2
48

Vinay's response is correct, with one minor addition.

Django's unit test framework actually sets DEBUG to False when it runs, so no matter what you have in settings.py, you will not have anything populated in connection.queries in your unit test unless you re-enable debug mode. The Django docs explain the rationale for this as:

Regardless of the value of the DEBUG setting in your configuration file, all Django tests run with DEBUG=False. This is to ensure that the observed output of your code matches what will be seen in a production setting.

If you're certain that enabling debug will not affect your tests (such as if you're specifically testing DB hits, as it sounds like you are), the solution is to temporarily re-enable debug in your unit test, then set it back afterward:

def test_myself(self):
    from django.conf import settings
    from django.db import connection

    settings.DEBUG = True
    connection.queries = []

    # Test code as normal
    self.assert_(connection.queries)

    settings.DEBUG = False
8
  • 3
    Thanks. Toggling DEBUG is precisely what I needed to do. :) Aug 10, 2009 at 13:04
  • 12
    An additional note: you should really wrap your test code in a try: block, putting settings.DEBUG = False in a corresponding finally: block. This way your other tests won't get "tainted" by the DEBUG setting if this one fails. Apr 28, 2010 at 4:22
  • 2
    you can use connection.use_debug_cursor = True instead of settings.DEBUG = True. On my opinion it will be more local solution
    – Oduvan
    Jun 24, 2012 at 12:39
  • why wouldn't you just put settings.DEBUG = True in the setUp() method, and settings.DEBUg = False in the tearDown() method?
    – stantonk
    Oct 25, 2012 at 5:01
  • 3
    This is a very bad way to change the settings (as hinted by @SmileyChris), Django has a whole bunch of ways to temporarily change settings without contaminating the other tests (and at least the @override_settings decorator has existed since Django 1.4)
    – Izkata
    May 25, 2018 at 16:59
31

If you are using pytest, pytest-django has django_assert_num_queries fixture for this purpose:

def test_queries(django_assert_num_queries):
    with django_assert_num_queries(3):
        Item.objects.create('foo')
        Item.objects.create('bar')
        Item.objects.create('baz')
11

If you don't want use TestCase (with assertNumQueries) or change settings to DEBUG=True, you can use context manager CaptureQueriesContext (same as assertNumQueries using).

from django.db import ConnectionHandler
from django.test.utils import CaptureQueriesContext

DB_NAME = "default"  # name of db configured in settings you want to use - "default" is standard
connection = ConnectionHandler()[DB_NAME]
with CaptureQueriesContext(connection) as context:
    ... # do your thing
num_queries = context.initial_queries - context.final_queries
assert num_queries == expected_num_queries

db settings

1
  • 4
    CaptureQueriesContext is a vastly under-valued test context handler. You can dig into all sorts of things about what the ORM did and why.
    – GDorn
    Feb 7, 2019 at 3:06
8

In modern Django (>=1.8) it's well documented (it's also documented for 1.7) here, you have the method reset_queries instead of assigning connection.queries=[] which indeed is raising an error, something like that works on django>=1.8:

class QueriesTests(django.test.TestCase):
    def test_queries(self):
        from django.conf import settings
        from django.db import connection, reset_queries

        try:
            settings.DEBUG = True
            # [... your ORM code ...]
            self.assertEquals(len(connection.queries), num_of_expected_queries)
        finally:
            settings.DEBUG = False
            reset_queries()

You may also consider resetting queries on setUp/tearDown to ensure queries are reset for each test instead of doing it on finally clause, but this way is more explicit (although more verbose), or you can use reset_queries in the try clause as many times as you need to evaluate queries counting from 0.

8

Here is the working prototype of context manager withAssertNumQueriesLessThan

import json
from contextlib import contextmanager
from django.test.utils import CaptureQueriesContext
from django.db import connections

@contextmanager
def withAssertNumQueriesLessThan(self, value, using='default', verbose=False):
    with CaptureQueriesContext(connections[using]) as context:
        yield   # your test will be run here
    if verbose:
        msg = "\r\n%s" % json.dumps(context.captured_queries, indent=4)
    else:
        msg = None
    self.assertLess(len(context.captured_queries), value, msg=msg)

It can be simply used in your unit tests for example for checking the number of queries per Django REST API call

    with self.withAssertNumQueriesLessThan(10):
        response = self.client.get('contacts/')
        self.assertEqual(response.status_code, 200)

Also you can provide exact DB using and verbose if you want to pretty-print list of actual queries to stdout

4

If you have DEBUG set to True in your settings.py (presumably so in your test environment) then you can count queries executed in your test as follows:

from django.db import connection

class DoSomethingTests(django.test.TestCase):
    def test_something_or_other(self):
        num_queries_old = len(connection.queries)
        do_something_in_the_database()
        num_queries_new = len(connection.queries)
        self.assertEqual(n, num_queries_new - num_queries_old)
4
  • Thanks. I tried it out but strangely enough len(connection.queries) is turning out to be Zero(!) before and after the call to the function. I tested it after replacing the function call with a straightforward call to MyModel.objects.filter() and still no luck. FYI I am using Django 1.1. Aug 10, 2009 at 11:33
  • Update: the mechanism works if I execute the function interactively using iPython. Of course this is against the development database as opposed to the transient test database. Does the discrepancy have something to do with the way Django executes tests in a transaction? Aug 10, 2009 at 11:38
  • 2
    DEBUG is by default set to False in django tests. This is because you want to test your live environment.
    – DylanYoung
    Oct 12, 2016 at 14:50
  • or: with self.settings(DEBUG=True): ... num_queries = len(connection.queries) see docs
    – djvg
    Jul 13, 2021 at 16:08
-1

If you want to use a decorator for that there is a nice gist:

import functools
import sys
import re
from django.conf import settings
from django.db import connection

def shrink_select(sql):
    return re.sub("^SELECT(.+)FROM", "SELECT .. FROM", sql)

def shrink_update(sql):
    return re.sub("SET(.+)WHERE", "SET .. WHERE", sql)

def shrink_insert(sql):
    return re.sub("\((.+)\)", "(..)", sql)

def shrink_sql(sql):
    return shrink_update(shrink_insert(shrink_select(sql)))

def _err_msg(num, expected_num, verbose, func=None):
    func_name = "%s:" % func.__name__ if func else ""
    msg = "%s Expected number of queries is %d, actual number is %d.\n" % (func_name, expected_num, num,)
    if verbose > 0:
        queries = [query['sql'] for query in connection.queries[-num:]]
        if verbose == 1:
            queries = [shrink_sql(sql) for sql in queries]
        msg += "== Queries == \n" +"\n".join(queries)
    return msg


def assertNumQueries(expected_num, verbose=1):

    class DecoratorOrContextManager(object):
        def __call__(self, func):  # decorator
            @functools.wraps(func)
            def inner(*args, **kwargs):
                handled = False
                try:
                    self.__enter__()
                    return func(*args, **kwargs)
                except:
                    self.__exit__(*sys.exc_info())
                    handled = True
                    raise
                finally:
                    if not handled:
                        self.__exit__(None, None, None)
            return inner

        def __enter__(self):
            self.old_debug = settings.DEBUG
            self.old_query_count = len(connection.queries)
            settings.DEBUG = True

        def __exit__(self, type, value, traceback):
            if not type:
                num = len(connection.queries) - self.old_query_count
                assert expected_num == num, _err_msg(num, expected_num, verbose)
            settings.DEBUG = self.old_debug

    return DecoratorOrContextManager()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.