34

I'm trying to use MongoDB to analyse Apache log files. I've created a receipts collection from the Apache access logs. Here's an abridged summary of what my models look like:

db.receipts.findOne()
{
    "_id" : ObjectId("4e57908c7a044a30dc03a888"),
    "path" : "/videos/1/show_invisibles.m4v",
    "issued_at" : ISODate("2011-04-08T00:00:00Z"),
    "status" : "200"
}

I've written a MapReduce function that groups all data by the issued_at date field. It summarizes the total number of requests, and provides a breakdown of the number of requests for each unique path. Here's an example of what the output looks like:

db.daily_hits_by_path.findOne()
{
    "_id" : ISODate("2011-04-08T00:00:00Z"),
    "value" : {
        "count" : 6,
        "paths" : {
            "/videos/1/show_invisibles.m4v" : {
                "count" : 2
            },
            "/videos/1/show_invisibles.ogv" : {
                "count" : 3
            },
            "/videos/6/buffers_listed_and_hidden.ogv" : {
                "count" : 1
            }
        }
    }
}

How can I make the output look like this instead:

{
    "_id" : ISODate("2011-04-08T00:00:00Z"),
    "count" : 6,
    "paths" : {
        "/videos/1/show_invisibles.m4v" : {
            "count" : 2
        },
        "/videos/1/show_invisibles.ogv" : {
            "count" : 3
        },
        "/videos/6/buffers_listed_and_hidden.ogv" : {
            "count" : 1
        }
    }
}
0

7 Answers 7

15

It's not currently possible, but I would suggest voting for this case: https://jira.mongodb.org/browse/SERVER-2517.

1
7

Taking the best from previous answers and comments:

db.items.find().hint({_id: 1}).forEach(function(item) {
    db.items.update({_id: item._id}, item.value);
});

From http://docs.mongodb.org/manual/core/update/#replace-existing-document-with-new-document
"If the update argument contains only field and value pairs, the update() method replaces the existing document with the document in the update argument, except for the _id field."

So you need neither to $unset value, nor to list each field.

From https://docs.mongodb.com/manual/core/read-isolation-consistency-recency/#cursor-snapshot "MongoDB cursors can return the same document more than once in some situations. ... use a unique index on this field or these fields so that the query will return each document no more than once. Query with hint() to explicitly force the query to use that index."

9
  • This leads to serious race conditions. Commented Feb 27, 2018 at 20:51
  • @DerekBrown, please explain or provide a proof or point to the bug in this code Commented Mar 2, 2018 at 10:37
  • db.items.update() is asynchronous, so the execution of the external call (ie db.items.find().hint()....foreach()) can terminate before all of the updates have been performed. Commented Mar 2, 2018 at 20:10
  • @DerekBrown, the code above uses standard synchronous JS API. If you use asynchronous driver, then it is your responsibility to wait for all async operations to complete. If you wanted to say that update is eventually consistent (which is not the same as asynchronous), and default write concern is not enough for your case, then please use more strict write concern configuration - docs.mongodb.com/manual/reference/write-concern Commented Mar 5, 2018 at 11:55
  • 1) The synchronous driver isn't officially supported. So at the very least you should mention that your code requires you use a different mongo driver. 2) Write Concern isn't relevant here. The problem isn't that the updates you have created aren't propagating to other shards, but that future queries (which require the updates be completed) have not yet been issued on the single instance that NodeJS is interacting with. Commented Mar 5, 2018 at 14:06
5

AFAIK, by design Mongo's map reduce will spit results out in "value tuples" and I haven't seen anything that will configure that "output format". Maybe the finalize() method can be used.

You could try running a post-process that will reshape the data using

results.find({}).forEach( function(result) {
  results.update({_id: result._id}, {count: result.value.count, paths: result.value.paths})
});

Yep, that looks ugly. I know.

1
  • is there not a way to modify the result object/document directly? Commented Oct 13, 2011 at 1:28
4

You can do Dan's code with a collection reference:

    function clean(collection) { 
      collection.find().forEach( function(result) {
      var value = result.value;
      delete value._id;     
      collection.update({_id: result._id}, value);     
      collection.update({_id: result.id}, {$unset: {value: 1}} ) } )};
3

A similar approach to that of @ljonas but no need to hardcode document fields:

db.results.find().forEach( function(result) {
    var value = result.value;
    delete value._id;
    db.results.update({_id: result._id}, value);
    db.results.update({_id: result.id}, {$unset: {value: 1}} )
} );
0
3

All the proposed solutions are far from optimal. The fastest you can do so far is something like:

var flattenMRCollection=function(dbName,collectionName) {
    var collection=db.getSiblingDB(dbName)[collectionName];

    var i=0;
    var bulk=collection.initializeUnorderedBulkOp();
    collection.find({ value: { $exists: true } }).addOption(16).forEach(function(result) {
        print((++i));
        //collection.update({_id: result._id},result.value);

        bulk.find({_id: result._id}).replaceOne(result.value);

        if(i%1000==0)
        {
            print("Executing bulk...");
            bulk.execute();
            bulk=collection.initializeUnorderedBulkOp();
        }
    });
    bulk.execute();
};

Then call it: flattenMRCollection("MyDB","MyMRCollection")

This is WAY faster than doing sequential updates.

1
  • 3
    NB: this is new in MongoDB 2.6
    – Vincent
    Commented Aug 11, 2014 at 1:26
0

While experimenting with Vincent's answer, I found a couple of problems. Basically, if you perform updates within a foreach loop, this will move the document to the end of the collection and the cursor will reach that document again (example). This can be circumvented if $snapshot is used. Hence, I am providing a Java example below.

final List<WriteModel<Document>> bulkUpdate = new ArrayList<>();

// You should enable $snapshot if performing updates within foreach
collection.find(new Document().append("$query", new Document()).append("$snapshot", true)).forEach(new Block<Document>() {
    @Override
    public void apply(final Document document) {
        // Note that I used incrementing long values for '_id'. Change to String if
        // you used string '_id's
        long docId = document.getLong("_id");
        Document subDoc = (Document)document.get("value");
        WriteModel<Document> m = new ReplaceOneModel<>(new Document().append("_id", docId), subDoc);
        bulkUpdate.add(m);

        // If you used non-incrementing '_id's, then you need to use a final object with a counter.
        if(docId % 1000 == 0 && !bulkUpdate.isEmpty()) {
            collection.bulkWrite(bulkUpdate);
            bulkUpdate.removeAll(bulkUpdate);
        }
    }
});
// Fixing bug related to Vincent's answer.
if(!bulkUpdate.isEmpty()) {
    collection.bulkWrite(bulkUpdate);
    bulkUpdate.removeAll(bulkUpdate);
}

Note : This snippet takes an average of 7.4 seconds to execute on my machine with 100k records and 14 attributes (IMDB dataset). Without batching, it takes an average of 25.2 seconds.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.