69

Is it possible to find the largest document size in MongoDB?

db.collection.stats() shows average size, which is not really representative because in my case sizes can differ considerably.

13

10 Answers 10

108

You can use a small shell script to get this value.

Note: this will perform a full table scan, which will be slow on large collections.

let max = 0, id = null;
db.test.find().forEach(doc => {
    const size = Object.bsonsize(doc); 
    if(size > max) {
        max = size;
        id = doc._id;
    } 
});
print(id, max);
4
  • 2
    I assume this size is in bytes?
    – akki
    Oct 23, 2016 at 10:00
  • @akki, yes, bsonsize returns bytes value (according to mognodb docs Jun 27, 2017 at 8:45
  • 12
    Is there a way to NOT load every document to client to calc its size? Perhaps using aggregation somehow. Aug 24, 2017 at 13:18
  • 1
    Work! Very slow, but Work!!! Nov 9, 2021 at 17:27
28

Starting Mongo 4.4, the new aggregation operator $bsonSize returns the size in bytes of a given document when encoded as BSON.

Thus, in order to find the bson size of the document whose size is the biggest:

// { "_id" : ObjectId("5e6abb2893c609b43d95a985"), "a" : 1, "b" : "hello" }
// { "_id" : ObjectId("5e6abb2893c609b43d95a986"), "c" : 1000, "a" : "world" }
// { "_id" : ObjectId("5e6abb2893c609b43d95a987"), "d" : 2 }
db.collection.aggregate([
  { $group: {
    _id: null,
    max: { $max: { $bsonSize: "$$ROOT" } }
  }}
])
// { "_id" : null, "max" : 46 }

This:

  • $groups all items together
  • $projects the $max of documents' $bsonSize
  • $$ROOT represents the current document for which we get the bsonsize
4
  • 1
    This solution worked fine for me - thanks! Aug 10, 2021 at 9:39
  • 2
    as of 2022, this is better than the accepted answer. Nov 21, 2022 at 15:11
  • 2
    How can you pull the document id? Nov 23, 2022 at 17:32
  • @AustinMutschler you should be able to work with the document with max size using a $top stage rather than a $group stage. Apr 30 at 13:59
26

Note: this will attempt to store the whole result set in memory (from .toArray) . Careful on big data sets. Do not use in production! Abishek's answer has the advantage of working over a cursor instead of across an in memory array.

If you also want the _id, try this. Given a collection called "requests" :

// Creates a sorted list, then takes the max
db.requests.find().toArray().map(function(request) { return {size:Object.bsonsize(request), _id:request._id}; }).sort(function(a, b) { return a.size-b.size; }).pop();

// { "size" : 3333, "_id" : "someUniqueIdHere" }
9
  • After running the accepted answer, this is the next script that anyone would want to run!
    – Mrchief
    Oct 14, 2014 at 17:31
  • 1
    I get an error running this: Error: assertion src\mongo\util\net\message_port.cpp:195 src/mongo/shell/query.js:113 Oct 16, 2015 at 6:33
  • 3
    This should not be the accepted answer. Calling toArray() on a large collection could crash the client. You can't pull 10 TB of data into the client's memory and then try to map it. You need to iterate it and let the driver handle batching. Dec 9, 2016 at 18:15
  • 2
    @PeteGarafano It says fairly clearly in the answer that it will pull it all into memory and is not for production. Dont down vote me because you copy and paste into prod.
    – Mike Graf
    Dec 9, 2016 at 21:54
  • 1
    Here's a much faster solution using using aggregation, which also doesn't require bringing the whole result set on the client. May 8, 2019 at 20:31
18

Finding the largest documents in a MongoDB collection can be ~100x faster than the other answers using the aggregation framework and a tiny bit of knowledge about the documents in the collection. Also, you'll get the results in seconds, vs. minutes with the other approaches (forEach, or worse, getting all documents to the client).

You need to know which field(s) in your document might be the largest ones - which you almost always will know. There are only two practical1 MongoDB types that can have variable sizes:

  • arrays
  • strings

The aggregation framework can calculate the length of each. Note that you won't get the size in bytes for arrays, but the length in elements. However, what matters more typically is which the outlier documents are, not exactly how many bytes they take.

Here's how it's done for arrays. As an example, let's say we have a collections of users in a social network and we suspect the array friends.ids might be very large (in practice you should probably keep a separate field like friendsCount in sync with the array, but for the sake of example, we'll assume that's not available):

db.users.aggregate([
    { $match: {
        'friends.ids': { $exists: true }
    }},
    { $project: { 
        sizeLargestField: { $size: '$friends.ids' } 
    }},
    { $sort: {
        sizeLargestField: -1
    }},
])

The key is to use the $size aggregation pipeline operator. It only works on arrays though, so what about text fields? We can use the $strLenBytes operator. Let's say we suspect the bio field might also be very large:

db.users.aggregate([
    { $match: {
        bio: { $exists: true }
    }},
    { $project: { 
        sizeLargestField: { $strLenBytes: '$bio' } 
    }},
    { $sort: {
        sizeLargestField: -1
    }},
])

You can also combine $size and $strLenBytes using $sum to calculate the size of multiple fields. In the vast majority of cases, 20% of the fields will take up 80% of the size (if not 10/90 or even 1/99), and large fields must be either strings or arrays.


1 Technically, the rarely used binData type can also have variable size.

3
  • 5
    @Sammaye the code in the accepted answer executes by client shell, not by databse server. So it loads to the client every document one by one using cursor, calculates it's size, and determines the largest one. Yes, it doesn't store the full collection in client's memory, but it still needs to transfer data via network. So that approach isn't quite usefull for large collections. Jun 23, 2019 at 20:19
  • 1
    @Sammaye I'm not exactly sure how aggregation transfers data between shards, but in single server configuration it's a good point if you can avoid transfering the full collection and calculate everything on the server. Even though if db loads all the data into it's memory. Jun 23, 2019 at 20:33
  • One thing to note would be that much of the speedup is likely coming from the client/server round trip where the client is your laptop and the server is a remote db host across WAN. So it would be interesting (i suspect negligible) how must faster it would be when running other client scripted vs agg framework solutions on the mongodb host (ie you connect to the db on localhost) ...
    – Mike Graf
    Jul 9, 2021 at 19:39
3

Well.. this is an old question.. but - I thought to share my cent about it

My approach - use Mongo mapReduce function

First - let's get the size for each document

db.myColection.mapReduce
(
   function() { emit(this._id, Object.bsonsize(this)) }, // map the result to be an id / size pair for each document
   function(key, val) { return val }, // val = document size value (single value for each document)
   { 
       query: {}, // query all documents
       out: { inline: 1 } // just return result (don't create a new collection for it)
   } 
)

This will return all documents sizes although it worth mentioning that saving it as a collection is a better approach (the result is an array of results inside the result field)

Second - let's get the max size of document by manipulating this query

db.metadata.mapReduce
(
    function() { emit(0, Object.bsonsize(this))}, // mapping a fake id (0) and use the document size as value
    function(key, vals) { return Math.max.apply(Math, vals) }, // use Math.max function to get max value from vals (each val = document size)
    { query: {}, out: { inline: 1 } } // same as first example
)

Which will provide you a single result with value equals to the max document size

In short:

you may want to use the first example and save its output as a collection (change out option to the name of collection you want) and applying further aggregations on it (max size, min size, etc.)

-OR-

you may want to use a single query (the second option) for getting a single stat (min, max, avg, etc.)

2
  • 2
    Your second example worked great for small data sets but actually managed to take my (hosted) server down with an out-of-memory error when run on a collection with about 400k documents. What is consuming all the memory? Can it really just not handle producing an array with 400k elements to pass to the reduce function? The doc says it should handle up to half Mongo's 16MB limit as a value argument to emit and as input to reduce. This should only be returning 8 bytes per emit! Shouldn't it call reduce multiple times if it nears memory limits? What is going on here?
    – Derek
    Aug 27, 2019 at 16:39
  • That's an important benchmark - thank you for sharing this data with us! To your question: I think it depends on 2 factors - first: available resources on hosted machine and second - the amount of load this machine handled when you executed your query (was it the only think handled by Mongo?). Nevertheless, this behavior is far from being accepted - you shouldn't be able to "take down" your server by performing such an activity. I find this disturbing. In person,I think this comment should turn into a question (probably at serverfault.com)
    – ymz
    Aug 28, 2019 at 6:44
3

If you want to get more details about the records (i.e. documents) which has highest size

  1. Calculate size
  2. Sort basis calculated size
  3. Limit to your desired number of records
  4. (Optionaly) Project to get only the fields which you want.
db.mycollection.aggregate([
    { $addFields: {
        bsonsize: { $bsonSize: "$$ROOT" }
    }},
    { $sort: { bsonsize: -1 }},
    { $limit: 10 },
    { $project: {
        _id: 1,
        bsonsize: 1
    }}
])

Note: (Performance)
This will scan entire collection. If you have vert large collection, then it can consume lot of IOPS. Run when database is at lowest activity level.

My collection contained 85 Million Records and its size was 30GB. It took 75 seconds to run.

2

If you're working with a huge collection, loading it all at once into memory will not work, since you'll need more RAM than the size of the entire collection for that to work.

Instead, you can process the entire collection in batches using the following package I created: https://www.npmjs.com/package/mongodb-largest-documents

All you have to do is provide the MongoDB connection string and collection name. The script will output the top X largest documents when it finishes traversing the entire collection in batches.

Preview

5
  • 1
    This is exactly what the built in cursor allows for. It streams the data rather than storing the entire collection to ram.
    – dmo
    Aug 31, 2017 at 23:31
  • Hi @dmo, could you please provide a command to achieve this via the built-in cursor?
    – Elad Nava
    Sep 6, 2017 at 5:06
  • 1
    collection.find() returns a cursor. The cursor is a stream of data. So in JS you can do something like this... jsfiddle.net/ro6efkdz
    – dmo
    Sep 11, 2017 at 16:39
  • @dmo: how does that cursor.on('data', ...) approach compare with the accepted answer? Is it any faster? Does it consume any less memory? May 8, 2019 at 0:03
  • This answer doesn't make much sense, the default cursor of any client driver would not load the collection into memory, in fact if you were to go down the aggregation framework route as Dan mentioned, then it would load that entire result set into memory. It is good to note that does exactly the same as the accepted answer, only in node.js github.com/eladnava/mongodb-largest-documents/blob/master/lib/…
    – Sammaye
    May 8, 2019 at 22:08
-1

Inspired by Elad Nana's package, but usable in a MongoDB console :

function biggest(collection, limit=100, sort_delta=100) {
  var documents = [];
  cursor = collection.find().readPref("nearest");
  while (cursor.hasNext()) {
    var doc = cursor.next();
    var size = Object.bsonsize(doc);
    if (documents.length < limit || size > documents[limit-1].size) {
      documents.push({ id: doc._id.toString(), size: size });
    }
    if (documents.length > (limit + sort_delta) || !cursor.hasNext()) {
      documents.sort(function (first, second) {
        return second.size - first.size;
      });
      documents = documents.slice(0, limit);
    }
  }
  return documents;
}; biggest(db.collection)
  • Uses cursor
  • Gives a list of the limit biggest documents, not just the biggest
  • Sort & cut output list to limit every sort_delta
  • Use nearest as read preference (you might also want to use rs.slaveOk() on the connection to be able to list collections if you're on a slave node)
-1

As Xavier Guihot already mentioned, a new $bsonSize aggregation operator was introduced in Mongo 4.4, which can give you the size of the object in bytes. In addition to that just wanted to provide my own example and some stats.

Usage example:

// I had an `orders` collection in the following format
[
  {
    "uuid": "64178854-8c0f-4791-9e9f-8d6767849bda",
    "status": "new",
    ...
  },
  {
    "uuid": "5145d7f1-e54c-44d9-8c10-ca3ce6f472d6",
    "status": "complete",
    ...
  },
  ...
];

// and I've run the following query to get documents' size
db.getCollection("orders").aggregate(
  [
    {
      $match: { status: "complete" } // pre-filtered only completed orders
    },
    {
      $project: {
        uuid: 1,
        size: { $bsonSize: "$$ROOT" } // added object size
      }
    },
    {
      $sort: { size: -1 }
    },
  ],
  { allowDiskUse: true } // required as I had huge amount of data
);

as a result, I received a list of documents by size in descending order.

Stats:

For the collection of ~3M records and ~70GB size in total, the query above took ~6.5 minutes.

-1

Here is one possible solution

db.collection.aggregate([
  {
    $addField: {
      objectSize: { $bsonSize: "$$ROOT" },
    },
  },
  {
    $sort: {
      objectSize: -1,
    },
  },
  {
    $limit: 1,
  },
  {
    $project: {
      objectSize: 0,
    },
  },
]);

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.