34

I have two mongo collections, one which refers to the production env and the other to the testing env.

How can I compare a diff between my two collections?

I tried dumping them to a bson and then converting to a json. But I cant just perform a simple diff on them since the sorting might vary and the json file is way too large to be sorted.

7 Answers 7

17

Try the following in the shell, it will iterate each item within a collection and try to match each document based on ID.

Say we have 2 collections db.col1 and db.col2:

> db.col1.find()
{ "_id" : 1, "item" : 1 }
{ "_id" : 2, "item" : 2 }
{ "_id" : 3, "item" : 3 }
{ "_id" : 4, "item" : 4 }

> db.col2.find()
{ "_id" : 1, "item" : 1 }
{ "_id" : 2, "item" : 2 }
{ "_id" : 3, "item" : 3 }
{ "_id" : 4, "item" : 4 }

We can then create a javascript function to compare 2 collections

function compareCollection(col1, col2){
    if(col1.count() !== col2.count()){
        return false;
    }

    var same = true;

    var compared = col1.find().forEach(function(doc1){
        var doc2 = col2.findOne({_id: doc1._id});

        same = same && JSON.stringify(doc1)==JSON.stringify(doc2);
    });

    return same;
}

Then call is like the following:

> compareCollection(db.col1, db.col2)
true

If we then have a 3rd collections db.col3

> db.col3.find()
{ "_id" : 1, "item" : 1 }

And compare this one

> compareCollection(db.col1, db.col3)
false

we'll get the expected result.

If we also have a 4th collection which has matching documents but diffrent data db.col4

> db.col4.find()
{ "_id" : 1, "item" : 10 }
{ "_id" : 2, "item" : 2 }
{ "_id" : 3, "item" : 3 }
{ "_id" : 4, "item" : 4 }

This will also return false

> compareCollection(db.col1, db.col4)
false
3
  • 2
    What if I've got 5 million records? Wouldnt that result in some performance issues? Isnt there a straighter way?
    – C4d
    Commented May 27, 2019 at 22:20
  • 3
    If you have 5 million records... the only way to compare them is to check all 5 million, I guess if you had something else in your document that made them unique you could just compare against that and have that as an index, but if you need to check that they match you need to check that they match 😕 Commented Jun 3, 2019 at 11:49
  • Mingo.io has a feature that does exactly what you need.
    – Radko
    Commented Feb 16, 2020 at 7:38
13

Starting Mongo 4.4, the aggregation framework provides a new $unionWith stage, performing the union of two collections (the combined pipeline results from two collections into a single result set).

Making it way easier to find the diff between two collections:

// > db.test.find()
//    { "a" : 9, "b" : 2  }
//    { "a" : 4, "b" : 12 }
//    { "a" : 3, "b" : 5  }
//    { "a" : 0, "b" : 7  }
//    { "a" : 7, "b" : 12 }
// > db.prod.find()
//    { "a" : 3, "b" : 5  }
//    { "a" : 4, "b" : 12 }
//    { "a" : 3, "b" : 5  }
//    { "a" : 0, "b" : 7  }
db.test.aggregate(
  { $unset: "_id" },
  { $project: { from: "test", doc: "$$ROOT" } },
  { $unionWith: {
      coll: "prod",
      pipeline: [
        { $unset: "_id" },
        { $project: { from: "prod", doc: "$$ROOT" } }
      ]
  }},
  { $group: {
      _id: "$doc",
      test: { $sum: { $cond: [ { $eq: ["$from", "test"] }, 1, 0 ] } },
      prod: { $sum: { $cond: [ { $eq: ["$from", "prod"] }, 1, 0 ] } }
  }},
  { $match: { $expr: { $ne: ["$test", "$prod"] } } }
)
// { "_id" : { "a" : 7, "b" : 12 }, "test" : 1, "prod" : 0 }
// { "_id" : { "a" : 9, "b" : 2  }, "test" : 1, "prod" : 0 }
// { "_id" : { "a" : 3, "b" : 5  }, "test" : 1, "prod" : 2 }

This:

  • $unsets the _id in order to latter be able to $group documents by themselves without considering the _id (as it might be different in the other collection).
  • $projects the field from whose value is the collection the document comes from (test or prod), in order, latter when we merge the two collections, to keep track from where documents originated from.
  • Also $projects the field doc whose value is the document itself (thanks to the $$ROOT variable). This is the field that will be used to $group documents together.
  • $unionWith the prod collection in order to merge documents from both collections into the same aggregation pipeline. The pipeline parameter is an optional aggregation pipeline applied on documents from the collection being merged (prod) before documents are inserted into the downstream pipeline. And we're applying the same $unset/$project stages we've applied on test documents.
  • $groups test and prod documents based on the doc field that we've created to represent the actual document. And we accumulate the two fields test and prod as the $sum (count) of grouped documents originating from one or the other collection (via $cond if expressions).
  • $matches resulting grouped elements by only keeping items not having the same number of test and prod documents: the actual diff between the two collections.
1
  • 1
    Will it work for array elements?
    – Gibbs
    Commented Mar 16, 2023 at 11:57
11

The dbHash has done the trick:

use db_name
db.runCommand('dbHash')

It returns the hash values for each collection. You can then compare them. It's pretty accurate.

2
  • 1
    It didn't return same hash for two collections with the same data. Commented Jun 12, 2018 at 14:44
  • 13
    _id will always lead to hash mismtach
    – VGupta
    Commented Jul 2, 2018 at 20:49
10

Using the Kevin Smith response, I have a new version, only for compare and return those ID's who the collectionB don't have comparing with the collectionA. And save the result in a collectionC when you have lots records.

    db.collectionA.find().forEach(function(doc1){
        var doc2 = db.collectionB.findOne({_id: doc1._id});
        if (!(doc2)) {
                db.collectionC.insert(doc1);
        }
    });
3
  • 1
    This is the best, most concise answer, thank you Carlos. Note: i had to delete doc1.__v and doc2.__v before stringifying the two docs when comparing them. Commented Jun 8, 2020 at 17:47
  • @chichilatte How did you delete the variable?
    – Mallik
    Commented Oct 4, 2020 at 8:45
  • @Mallik just with a line like delete doc1.__v inside that forEach callback function Commented Oct 6, 2020 at 17:13
5

If you need to compare only a subset of fields (e.g. you don't need to compare ids), you can do it the following way. Export the collections to csv, specifying the fields to compare (source):

mongoexport -d <db_name> -c <col_name> --fields "field1,field2" --type=csv | sort > export.csv

And then do a simple diff on the csv files. Note that column order in the csv file corresponds to the --field option.

Pros:

  • you can specify subset of fields to compare.
  • you can see the actual diff of the records.

Cons:

  • to compare the full records, you need to know all possible fields.
  • mongoexport can be slow for huge databases.

To get all fields in all documents in a collection, see this answer.

1

mongoexport now has a --sort option:

For example:

$ mongo
test> db.coll.insertMany([
  { _id: 0, name: 'Alex' },
  { _id: 1, name: 'Bart' },
  { _id: 2, name: 'Maria' },
  { _id: 3, name: 'Aristotle' },
]);
{
  "acknowledged": true,
  "insertedIds": [
    0,
    1,
    2,
    3
  ]
}

Export it:

mongoexport -d test -c coll --sort "{name: 1}"
2018-10-25T15:50:07.210+0300    connected to: localhost
{"_id":0.0,"name":"Alex"}
{"_id":3.0,"name":"Aristotle"}
{"_id":1.0,"name":"Bart"}
{"_id":2.0,"name":"Maria"}
2018-10-25T15:50:07.210+0300    exported 4 records

mongoexport -d test -c coll --sort "{name: -1}"
2018-10-25T15:49:42.010+0300    connected to: localhost
{"_id":2.0,"name":"Maria"}
{"_id":1.0,"name":"Bart"}
{"_id":3.0,"name":"Aristotle"}
{"_id":0.0,"name":"Alex"}
2018-10-25T15:49:42.011+0300    exported 4 records

After you exported the collections you can perform a diff in the command line or use a graphical user interface (GUI) tool like Beyond Compare.

FYI: if your documents might have the same data but have different _id values: You can exclude the _id field when exporting like this: https://stackoverflow.com/a/49895549/728287

0

use Studio 3T to compare mongodb. you can compare collections,db,single records as well. just you need to download and connect mongo. here is download link https://studio3t.com/

2
  • Maybe you might add some link to some tutorial on how to do it in that specific tool.
    – testuser
    Commented Nov 29, 2017 at 12:52
  • 2
    Especially because this feature is not a part of the free edition (not even of the "cheap" license), but only of the Enterprise edition... If someone does have the tool though, here's the link to the real tutorial: studio3t.com/whats-new/diff-mongodb/…
    – testuser
    Commented Nov 29, 2017 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.