1

I am looping over 13,000 in-memory city names and generating queries to filter for something. I've encountered something I cannot explain...

When the loop has a single line:

cities = City.objects.filter(name__iexact=city)

performance is almost 800 items/second

When the loop measures the length of the returned set...

cities = City.objects.filter(name__iexact=city)
num_citles = len(cities)

performance drops to 8 items/second

I can't explain where the performance degradation occurrs. Obviously, I'm missing something... Why would counting the number of items on an in-memory array that is always between 0 and 3 items reducing performance by a factor of x100?

1

1 Answer 1

1

Django querysets are lazy so QuerySet.filter does not actually evaluate the queryset i.e. run the queries in the database. When you run len function on it, it is evaluated and it will get all the items from database after running the filter only to get the count. Hence, the count is very slower.

You'll get a far better performance if you run COUNT on the database level:

num_cities = City.objects.filter(name__iexact=city).count()
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.