0

Ruby Mongo Driver question:

How do I output 5_000 document batches from the collection at a time until I read the last document in the collection without dumping the entire database into memory first?

This is really bad method for me:

mongo = MongoClient.new('localhost', 27017)['sampledb']['samplecoll']
@whois.find.to_a....
1
  • If I can understand you don't want to load entire bunch of records? Use limit and skip then. Jan 31, 2013 at 6:20

1 Answer 1

1

Mongo::Collection#find returns a Mongo::Cursor that is Enumerable. For batch processing Enumerable#each_slice is your friend and well worth adding to your toolkit.

Hope that you like this.

find_each_slice_test.rb

require 'mongo'
require 'test/unit'

class FindEachSliceTest < Test::Unit::TestCase
  def setup
    @samplecoll = Mongo::MongoClient.new('localhost', 27017)['sampledb']['samplecoll']
    @samplecoll.remove
  end

  def test_find_each_slice
    12345.times{|i| @samplecoll.insert( { i: i } ) }
    slice__max_size = 5000
    @samplecoll.find.each_slice(slice__max_size) do |slice|
      puts "slice.size: #{slice.size}"
      assert(slice__max_size >= slice.size)
    end
  end
end

ruby find_each_slice_test.rb

Run options: 

# Running tests:

slice.size: 5000
slice.size: 5000
slice.size: 2345
.

Finished tests in 6.979301s, 0.1433 tests/s, 0.4298 assertions/s.

1 tests, 3 assertions, 0 failures, 0 errors, 0 skips

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.