6

Some of my unit tests take 10-15 seconds just for mysql to create the tables. This seems unnecessarily long. It has to create around 50 tables, but that's still only 3 tables per second. This is a big annoyance when running unit tests over-and-over.

As a workaround, I have been running my unit tests in sqlite3. It is blazing fast, but I would prefer to run my tests on MySQL since that's what my live servers run.

To illustrate the speed difference, create a fresh project. Then run syncdb on it using mysql. Then try it using sqlite3.

[~/testproject] ./manage.py syncdb
Creating table auth_permission
Creating table auth_group
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site

For me, it takes about 2 seconds to create the above tables in MySQL. Sqlite3 is almost instant.

I am running mysql on my development machine. Here is my my.cnf.

Please suggest any tips or tweaks you can think of that might help speed up MySQL's table creation time.

7
  • Is your MySQL table local or remote?
    – Martin
    Jan 8, 2010 at 2:11
  • It is local on my mac. Check out my my.cnf if you like: pastebin.com/m69af8ba3
    – Gattster
    Jan 8, 2010 at 2:13
  • 1
    "I would prefer to run my tests on MySQL" Why? You don't trust the Django ORM layer?
    – S.Lott
    Jan 8, 2010 at 2:15
  • I have some code that executes raw sql for performance. The raw sql only works in mysql. There are other valid use cases, like if you used mysql full text search.
    – Gattster
    Jan 8, 2010 at 2:22
  • 1
    @andybak - I'm fairly certain there's no way other than a new settings.py. It'd be nice if Django had an option for specifying a completely different database setup for tests, but it doesn't seem to. Jan 8, 2010 at 11:51

4 Answers 4

3

You can create RAM-disk and move db there, just for unit testing. If you write script for this then it's automatic and very convenient.

Also, for other purposes I've written custom test runner that loads DB from sql dump instead of creating it and then creating tables.

You choose.

4
  • I've never ran mysql on a ram disk, but I assume it's as simple as setting up a new mysql daemon on a different port, and pointing its data dir to a ram disk. Thanks for the tips.
    – Gattster
    Jan 8, 2010 at 18:35
  • Yes, you could also just switch config (to the one with db path pointing to ram disk) for your existing mysql instance and then restart it - whichever works better for you. Jan 8, 2010 at 18:57
  • Have you actually ran mysql on a ram disk and seen a speed improvement? It seems like it would help but until you test it you really don't know.
    – Gattster
    Jan 8, 2010 at 20:38
  • Yes, I've done it myself and experienced major speed improvement (although I haven't measured times to write exact numbers). Jan 8, 2010 at 21:57
1

I've been experiencing slow INNODB table creation speeds (about 25 seconds to create 13 small tables).

I experimented with options in the [mysqld] section of the my.cnf file.

Adding:

innodb_flush_method=fdatasync

produced the best results (about 1.5 seconds to create the same 13 small tables).

1

I have found that using sqlite as a replacement makes my unit tests much faster. I'm also removing southdb, as this slows down table creation too.

if len(sys.argv) > 1 and sys.argv[1] == 'test':
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.sqlite3',
            'NAME': ':memory',
            'USER': '',
            'PASSWORD': '',
            'HOST': '',
            'PORT': '',
        }
    }
    INSTALLED_APPS = tuple([x for x in INSTALLED_APPS if x != 'south'])
1
  • This is kind of a hack, and might produce unexpected results when doing advanced database stuff (in other words, your test might pass under sqlite when it would fail under mysql). That sid, this is a good solution to get going on test development quickly when you need mysql for your production server. Mar 28, 2012 at 0:31
0

On Mac OSX only, add the following to your ~/.my.cnf:

[mysqld]
skip-sync-frm=ON

For me, this improved the startup time of my Django test suite on MySQL from 1m30s to 7s!

Details here: http://www.stereoplex.com/blog/speeding-up-django-unit-test-runs-with-mysql

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.