234

How would I find duplicate fields in a mongo collection.

I'd like to check if any of the "name" fields are duplicates.

{
    "name" : "ksqn291",
    "__v" : 0,
    "_id" : ObjectId("540f346c3e7fc1054ffa7086"),
    "channel" : "Sales"
}

Many thanks!

0

10 Answers 10

387

Use aggregation on name and get name with count > 1:

db.collection.aggregate([
    {"$group" : { "_id": "$name", "count": { "$sum": 1 } } },
    {"$match": {"_id" :{ "$ne" : null } , "count" : {"$gt": 1} } }, 
    {"$project": {"name" : "$_id", "_id" : 0} }
]);

To sort the results by most to least duplicates:

db.collection.aggregate([
    {"$group" : { "_id": "$name", "count": { "$sum": 1 } } },
    {"$match": {"_id" :{ "$ne" : null } , "count" : {"$gt": 1} } }, 
    {"$sort": {"count" : -1} },
    {"$project": {"name" : "$_id", "_id" : 0} }     
]);

To use with another column name than "name", change "$name" to "$column_name"

11
  • 2
    "$match": {"_id" :{ "$ne" : null } - is unnecessary here, since the second part of the statement would suffice filtering the result. So only checking for the group having count > 1 will do.
    – BatScream
    Nov 18, 2014 at 1:21
  • 7
    Tks @BatScream. { "$ne" : null } is there just in case 'name' is null or doesn't exist. Aggregation will count null as well.
    – anhlc
    Nov 18, 2014 at 1:30
  • 1
    Welcome. But then why check the _id field. It is always guaranteed to be not null after the group operation.
    – BatScream
    Nov 18, 2014 at 1:32
  • 4
    The _id of a document from a $group stage can be null.
    – wdberkeley
    Nov 18, 2014 at 8:35
  • 1
    What will be the output of this? If i run i get all the documents what i need is i want only the duplicated id's/names.
    – Kannan T
    Dec 18, 2017 at 6:38
46

You can find the list of duplicate names using the following aggregate pipeline:

  • Group all the records having similar name.
  • Match those groups having records greater than 1.
  • Then group again to project all the duplicate names as an array.

The Code:

db.collection.aggregate([
{$group:{"_id":"$name","name":{$first:"$name"},"count":{$sum:1}}},
{$match:{"count":{$gt:1}}},
{$project:{"name":1,"_id":0}},
{$group:{"_id":null,"duplicateNames":{$push:"$name"}}},
{$project:{"_id":0,"duplicateNames":1}}
])

o/p:

{ "duplicateNames" : [ "ksqn291", "ksqn29123213Test" ] }
3
  • 2
    The fact that you explain what each line does makes this answer optimal.
    – Zap
    Dec 7, 2020 at 10:48
  • How can I get the duplicate data, based on two fields. Basic example: let say I've collection in which I'm storing social details like: ``` [{username: 'abc', type: 'facebook'}, {username: 'abc', type: 'instagram'}] ``` so In that case I don't want only based on username, but based on both "username & type". thanks :) May 31, 2021 at 9:59
  • is there a way to use this in such a way that I see the complete documents that have duplicate names? Sep 6, 2023 at 12:37
21

Another option is to use $sortByCount stage.

db.collection.aggregate([
  { $sortByCount: '$name' }
]

This is the combination of $group & $sort.

The $sortByCount stage is equivalent to the following $group + $sort sequence:

    { $group: { _id: <expression>, count: { $sum: 1 } } },
    { $sort: { count: -1 } }
2
  • is $name the field name? also how can you change the sorting to sort by count descending? Thanks! Jan 24, 2022 at 16:03
  • 1
    To only get duplicates (as per the question), add a $match stage to the aggregation after the $sortByCount: {$match: {count: {$gt: 1}}}
    – M. Justin
    Apr 27, 2022 at 21:39
15

The answer anhic gave can be very inefficient if you have a large database and the attribute name is present only in some of the documents.

To improve efficiency you can add a $match to the aggregation.

db.collection.aggregate(
    {"$match": {"name" :{ "$ne" : null } } }, 
    {"$group" : {"_id": "$name", "count": { "$sum": 1 } } },
    {"$match": {"count" : {"$gt": 1} } }, 
    {"$project": {"name" : "$_id", "_id" : 0} }
)
0
14
db.getCollection('orders').aggregate([  
    {$group: { 
            _id: {name: "$name"},
            uniqueIds: {$addToSet: "$_id"},
            count: {$sum: 1}
        } 
    },
    {$match: { 
        count: {"$gt": 1}
        }
    }
])

First Group Query the group according to the fields.

Then we check the unique Id and count it, If count is greater then 1 then the field is duplicate in the entire collection so that thing is to be handle by $match query.

3
  • 1
    haven't been able to make this one work for me too. Down voting!
    – Mathieu G
    Dec 8, 2016 at 15:06
  • 1
    This post is old but may help some one . check this out I'll check in my local it's working. Even I came across one blog regarding this. Please have a look. compose.com/articles/finding-duplicate-documents-in-mongodb May 2, 2019 at 13:17
  • I was able to get it to work - edited to update to confirmed working version. Jul 15, 2020 at 5:21
5

enter image description here

this is how we can achieve this in mongoDB compass

2

In case you need to see all duplicated rows:

db.collection.aggregate([
     {"$group" : { "_id": "$name", "count": { "$sum": 1 },"data": { "$push": "$$ROOT" }}},
     {"$unwind": "$data"},
     {"$match": {"_id" :{ "$ne" : null } , "count" : {"$gt": 1} } }, 
]);
1
  • Error: Line 4: Unexpected token {
    – dcsan
    Dec 22, 2021 at 8:46
1

If somebody is looking for a query for duplicates with an extra "$and" where clause, like "and where someOtherField is true"

The trick is to start with that other $match, because after the grouping you don't have all the data available anymore

// Do a first match before the grouping
{ $match: { "someOtherField": true }},
{ $group: {
    _id: { name: "$name" },
    count: { $sum: 1 }
}},
{ $match: { count: { $gte: 2 } }},

I searched for a very long time to find this notation, hope I can help somebody with the same problem

1

Sometimes you want to find duplicates regardless the case, when you want to create a case insensitive index for instance. In this case you can use this aggregation pipeline

db.collection.aggregate([
  {'$group': {'_id': {'$toLower': '$name'}, 'count': { '$sum': 1 }, 'duplicates': { '$push': '$$ROOT' } } },
  {'$match': { 'count': { '$gt': 1 } } }
]);

Explanation:

  • group by name but first change the case to lower case and push the docs to the duplicates array.
  • match those groups having records greater than 1 (the duplicates).
0

Search for duplicates in Compass Mongo db using $sortByCount
[screenshot]: https://i.stack.imgur.com/L85QV.png

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.