61

Is there a simple way to remove duplicates in the following basic query:

email_list = Emails.objects.order_by('email')

I tried using duplicate() but it was not working. What is the exact syntax for doing this query without duplicates?

2

9 Answers 9

161

This query will not give you duplicates - ie, it will give you all the rows in the database, ordered by email.

However, I presume what you mean is that you have duplicate data within your database. Adding distinct() here won't help, because even if you have only one field, you also have an automatic id field - so the combination of id+email is not unique.

Assuming you only need one field, email_address, de-duplicated, you can do this:

email_list = Email.objects.values_list('email', flat=True).distinct()

However, you should really fix the root problem, and remove the duplicate data from your database.

Example, deleting duplicate Emails by email field:

for email in Email.objects.values_list('email', flat=True).distinct():
    Email.objects.filter(pk__in=Email.objects.filter(email=email).values_list('id', flat=True)[1:]).delete()

Or books by name:

for name in Book.objects.values_list('name', flat=True).distinct(): 
    Book.objects.filter(pk__in=Artwork.objects.filter(name=name).values_list('id', flat=True)[3:]).delete()
6
  • 1
    Great solution. When using .values(..) you could even pass that as kwargs to .filter(...)
    – vdboor
    Commented Apr 29, 2014 at 14:39
  • In the 2nd code example, we should set the varaible to delete all duplicates of Emails? cause once iteration finished, Email.objects become the whole queryset of Email objects, doesn't it? Commented Jul 21, 2016 at 10:40
  • 1
    In the second example [3:] looks odd to me Commented Oct 23, 2018 at 19:47
  • talking about solving the root cause, saved me from spending a lot of time! Thanks Commented Apr 22, 2021 at 17:05
20

For checking duplicate you can do a GROUP_BY and HAVING in Django as below. We are using Django annotations here.

from django.db.models import Count
from app.models import Email

duplicate_emails = Email.objects.values('email').annotate(email_count=Count('email')).filter(email_count__gt=1)

Now looping through the above data and deleting all other emails except the first one (depends on requirement or whatever).

for data in duplicates_emails:
    email = data['email']
    Email.objects.filter(email=email).order_by('pk')[1:].delete()
1
  • 2
    This seems not working: Cannot use 'limit' or 'offset' with delete. Replacing [1:] with .first() works. But this will only delete the first and not the rest if cnt > 2
    – cwhisperer
    Commented Jan 31, 2023 at 7:44
14

You can chain .distinct() on the end of your queryset to filter duplicates. Check out: http://docs.djangoproject.com/en/dev/ref/models/querysets/#django.db.models.query.QuerySet.distinct

1
  • 4
    You didn't understood the question. Commented Apr 18, 2021 at 5:09
8

You may be able to use the distinct() function, depending on your model. If you only want to retrieve a single field form the model, you could do something like:

email_list = Emails.objects.values_list('email').order_by('email').distinct()

which should give you an ordered list of emails.

3

You can also use set()

email_list = set(Emails.objects.values_list('email', flat=True))
2
  • I was trying to remove duplicates after creating a union of two querysets. I tried using the distinct method and setting all to false qs_1.union(qs_2, all=False). Couldn't get any of the above to work so used set. Commented Mar 15, 2019 at 23:29
  • Cautionary note: I had this issue and set() worked to remove the duplicated record, but it seems to break the order_by (whether I apply it before or afterward) Commented Jul 15, 2021 at 19:41
1

Use, self queryset.annotate()!

from django.db.models import Subquery, OuterRef

email_list = Emails.objects.filter(
    pk__in = Emails.objects.values('emails').distinct().annotate(
        pk = Subquery(
        Emails.objects.filter(
          emails= OuterRef("emails")
        )
        .order_by("pk")
        .values("pk")[:1])
    )
    .values_list("pk", flat=True)
)

This queryset goes to make this query.

 SELECT `email`.`id`,
        `email`.`title`,
        `email`.`body`,
       ...
       ...
  FROM `email`
 WHERE `email`.`id` IN (
        SELECT DISTINCT (
                SELECT U0.`id`
                  FROM `email` U0
                 WHERE U0.`email` = V0.`approval_status`
                 ORDER BY U0.`id` ASC
                 LIMIT 1
               ) AS `pk`
         FROM `agent` V0
 )

cheet-sheet

from django.db.models import Subquery, OuterRef

group_by_duplicate_col_queryset = Models.objects.filter(
    pk__in = Models.objects.values('duplicate_col').distinct().annotate(
        pk = Subquery(
        Models.objects.filter(
          duplicate_col= OuterRef('duplicate_col')
        )
        .order_by("pk")
        .values("pk")[:1])
    )
    .values_list("pk", flat=True)
)
0

I used the following to actually remove the duplicate entries from from the database, hopefully this helps someone else.

adds = Address.objects.all()
d = adds.distinct('latitude', 'longitude')
for address in adds:    
  if i not in d:
    address.delete()
2
  • 5
    Implementing loops around ORM operations is generally a bad idea, as it doesn't scale very well. In this example, you have many, many queries being executed. Suppose there are many rows returned in adds. In each loop, you launch a first query to see if i not in d, and possibly another one to delete the affected address records. You can do this in the ORM directly without the Python loop by doing something like: Address.objects.exclude(pk__in=d.values('pk, flat=True)).delete(). (You may need to tweak this -- I haven't tested).
    – BillyBBone
    Commented Feb 26, 2015 at 23:33
  • 1
    Great tip from BillyBBone. Adding a small correction to it. values() hasn't flat=True. Here values_list() should be used: Address.objects.exclude(pk__in=d.values_list('pk', flat=True)).delete(). Because flat=True will remove the tuples and return the list Commented Sep 7, 2018 at 9:31
0

if you want remove duplicacy from the queryset, for eg. let's say you have the user model with fields like name, email and you want remove duplicate emails then, you can simply use distinct() method

User.objects.all().distinct("email")

it will return all the unique emails.

-2

you can use this raw query : your_model.objects.raw("select * from appname_Your_model group by column_name")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.