1

this below snippet works just fine when used in java . but is there a way in mongodb mapreduce using java driver to store the results into a different database. i wanto achive this because mapreduce can not create capped collection. so i want to write the results of mapreduce to a different collection and create a capped collection in target database and insert into that collection.

`MapReduceCommand wktgtcmd = new MapReduceCommand(tgtCollection, wk_map_function, tgt_reduce_function, 
                    "collection String", MapReduceCommand.OutputType.MERGE, null);
tgtCollection.mapReduce(wktgtcmd);`

something like "database.Collection String" or some hints to achieve this?

2 Answers 2

2

Usually you can call the MapReduceCommand.setOutputDB method to tell the map reduce command which database to use.

However, after doing a quick test from the mongo shell, it looks like the mongodb server drops the collection first if it is capped.

If you want the results to go into a capped collection then the easiest way is to get the results inline and manually insert them into the capped collection yourself :

MapReduceCommand wktgtcmd = new MapReduceCommand(tgtCollection, 
                                                 wk_map_function, 
                                                 tgt_reduce_function, 
                                                 null,
                                                 MapReduceCommand.OutputType.INLINE, 
                                                 null);
 MapReduceOutput out = tgtCollection.mapReduce(cmd);


 DB db = mongo.getDB("database");
 DBCollection results = db.getCollection("collection");
 for (DBObject o : out.results()) {
    results.insert(o);
 }
2
  • is there any limitation to this approach? i read some where that the MapReduceOutput can only handle upto 16MB. i have data need to be written to table more than 100 MB. Jul 29, 2013 at 20:42
  • As you correctly point out, if you get the results inline then you will hit [mongoDB 16Mb document limitation](docs.mongodb.org/manual/reference/limits/#BSON Document Size). If the m/r result set is as large as you say, maybe it would be easier to set the output to a collection and then copy this result set into your capped collection afterwards. This might be slow depending on the m/r that you are running and the size of the results.
    – jimoleary
    Jul 31, 2013 at 10:36
0

Store the result by yourself. If it is over 16MB BSON limitation. Use GridFS!

https://docs.mongodb.com/manual/core/gridfs/

GridFS is a specification for storing and retrieving files that exceed the BSON-document size limit of 16 MB.

Instead of storing a file in a single document, GridFS divides the file into parts, or chunks [1], and stores each chunk as a separate document. By default, GridFS uses a chunk size of 255 kB; that is, GridFS divides a file into chunks of 255 kB with the exception of the last chunk. The last chunk is only as large as necessary. Similarly, files that are no larger than the chunk size only have a final chunk, using only as much space as needed plus some additional metadata.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.