0

I am writing a map-reduce function in MongoDB that will be used across multiple collections. The result of the map-reduce on each collection merges the results into the same output collection.

I would like:

  • To include the name of the collection that the document was from in the key so that the output collection contains a separate document if the same key was found in different source documents
  • The map function to reference the name of the collection without hard coding it so that four different map functions don't have to be written for four collections that contain the same logic otherwise (i.e. keep the map function DRY)

Is this possible?

More detail below...

map function:

function() {
    emit({ cpID: this.cpID, institutionID: this.institutionID, sourceCollection: collName }, 1);
};

reduce function:

function(key, values) {
    return values.length;
};

mongo shell commands:

db.loadServerScripts();
var collName;
collName = "activities";
db.activities.mapReduce(mapcpsinstitutions, reducecpsinstitutions, { out: {replace: "cpsAndSchools"}, "scope": { "collName": collName } } );
collName = "activitysummaries";
db.activitysummaries.mapReduce(mapcpsinstitutions, reducecpsinstitutions, { out: {reduce: "cpsAndSchools"}, "scope": { "collName": collName } } );
collName = "contentusagesummaries";
db.contentusagesummaries.mapReduce(mapcpsinstitutions, reducecpsinstitutions, { out: {reduce: "cpsAndSchools"}, "scope": { "collName": collName } } );
collName = "contentusages";
db.contentusages.mapReduce(mapcpsinstitutions, reducecpsinstitutions, { out: {reduce: "cpsAndSchools"}, "scope": { "collName": collName } } );

1 Answer 1

1

This seems a bit vague in your actual intent. But from the basic concept of keeping a singular "map" function that will use a "variable" name as part of the emitted "key", then you can always use the "scope" option for mapReduce:

var colname = "collection";

db.getCollection(colname).mapReduce(
    function() {

       var data = {};

       // Some map operations here

       emit( collection + idKey, data );

    },
    function(key,values) {
        var reduced = {};

        // Reduce operations here

       return reduced;
    },
    { 
        "out": { "merge": "newcollection" },
        "scope": { "collection": colname }
    }
)

So anything defined in "scope" is a "global" variable to the "map", "reduce" and "finalize" functions you can provide. Variables can be passed in this way without needing to alter the code of any of those functions.

The basic case here is to "pre-pend" the current collection name to the emitted key

4
  • Neil - first, your solution worked. However, this still requires me to set the variable explicitly each time I change the collection that I am performing the map-reduce on. I was wondering if there was a way to use 'this.' notation within the map function to avoid this declaration. (I will also add additional detail to my question above to show why this might be useful.) Commented Nov 6, 2014 at 15:13
  • @KushaTavakoli Ever heard of functions and parameters? As in just pass in colname as a parameter to a generic function.
    – Neil Lunn
    Commented Nov 6, 2014 at 15:15
  • ok, I think the underlying issue here that I can infer from your last response is that perhaps the fact that I am calling the function four times is a bit silly and I should really just create an array with the name of each collection I want to perform the map-reduce on and then loop through the array, using the collection name both in the variable declaration and in getCollection() as you have done above. This would be smarter. (Thank you!) Commented Nov 6, 2014 at 15:25
  • @KushaTavakoli That would sound logical. Try to use "async.each" if under node.js, or similar flow control.
    – Neil Lunn
    Commented Nov 6, 2014 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.