5

I have a table which I am working on and it contains 11 million rows there abouts... I need to run a migration on this table but since Django trys to store it all in cache I run out of ram or disk space which ever comes first and it comes to abrupt halt.

I'm curious to know if anyone has faced this issue and has come up with a solution to essentially "paginate" migrations maybe into blocks of 10-20k rows at a time?

Just to give a bit of background I am using Django 1.10 and Postgres 9.4 and I want to keep this automated still if possible (which I still think it can be)

Thanks Sam

1
  • Precisely what kind of migration is this? The dirty work of adding a new column/field with a default value should happen on the db backend, not within the django process running the migration. Did you modify the migration script?
    – rrauenza
    May 24, 2017 at 21:00

1 Answer 1

5

The issue comes from a Postgresql which rewrites each row on adding a new column (field).

What you would need to do is to write your own data migration in the following way:

  1. Add a new column with null=True. In this case data will not be rewritten and migration will finish pretty fast.
  2. Migrate it
  3. Add a default value
  4. Migrate it again.

That is basically a simple pattern on how to deal with adding a new row in a huge postgres database.

5
  • But is it possible to make a migration use this? May 24, 2017 at 20:09
  • what kind of migration are you talking about? Adding a row? Changing a format of data?
    – taras
    May 24, 2017 at 20:12
  • adding fields to the model May 24, 2017 at 20:17
  • I've updated the answer. Hope, you get at least an idea of where to go.
    – taras
    May 24, 2017 at 20:26
  • Unfortunately this didn't help as much as I would hoped, it certainly seems to be a limitation of Postgres like you said and instead I added more storage to my staging server and move the data directory of Postgres on production to give it access to a bigger drive. I will accept this answer as it provides some usual information still. Much appreciated. May 29, 2017 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.