5

With 2.2 we now have the option to bulk update: https://docs.djangoproject.com/en/3.0/ref/models/querysets/#bulk-update

I have a model with say millions of rows and I want to efficiently update all the records. I am trying to use bulk_update, but which means I still have to load all the model objects in the memory, modify the field one by one and then use bulk update:

What I am doing:

def migrate_event_ip_to_property(apps, schema_editor):
    Event = apps.get_model('app' 'Event')
    events = Event.objects.all()

    for event in events:
        if event.ip:
            event.properties["$ip"] = event.ip

    Event.objects.bulk_update(events, ['properties'], 10000)

Since there are millions of records, can I avoid doing Event.objects.all() and load all objects into the memory even while using bulk_update?

6
  • 1
    What is event.properties? You could run the bulk update on every 1000th iteration for example and use iterator() so that you don't load them all at once Apr 5, 2020 at 9:36
  • Also, how do you ensure that the order of objects is preserved in each iteration if you run it in batch? Also iterator does not save memory, generator does. Apr 5, 2020 at 9:48
  • A generator is an iterator btw Apr 5, 2020 at 10:02
  • What about restricting the fields you load in memory with Event.objects.all().values('id','properties') Apr 5, 2020 at 10:04

2 Answers 2

3
def migrate_event_ip_to_property(apps, schema_editor):
    Event = apps.get_model('app' 'Event')
    chunk = []
    # Use iterator to save memory
    for i, event in enumerate(Event.objects.only('properties', 'ip').iterator(chunk_size=10000)):
        if event.ip:
            event.properties['$ip'] = event.ip
            chunk.append(event)
        # Every 10000 events run bulk_update
        if i % 10000 == 0 and chunk:
            Event.objects.bulk_update(chunk, ['properties'])
            chunk = []
    if chunk:
        Event.objects.bulk_update(chunk, ['properties'])
0

If memory is your main concern, what about restricting the fields you load in memory with :

Event.objects.all().values('id','properties')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.