0

I've stumbled upon a rather strange problem. I have a mongo collection in my app with approximately 4 800 000 records. I am using a query that returns 2 400 000 records from this collection. Consequently I iterate over them and create embedded documents on each record. When I was looking through my logs I noticed that the iteration took place over 4 million times. I dug deeper and found out, that a lot of records got into the iteration multiple times. That means that a record with the same ID appeared as processed in the logs more than once. Below is the query:

MyModel.where({integer_array: {"$not" => {'$size' => 0}}}).all.each do |n|

Very strange is, that when I run in rails console MyModel.where({integer_array: {"$not" => {'$size' => 0}}}).count I get circa 2400000 as a result. I am running on rails 3.2.13, mongoid 3.0.21 and mongodb 2.4.0.

1 Answer 1

2

Cursors in mongodb are latent by default. This means that if further writes come in while you are processing the cursor then you may or may not see the results of these updates.

Baiscally as more data is added or existing data is modified the object may move around the collection which may causes them to appear in the cursor multiple times(the actual behaviour is unspecified).

You can use pass mongoid the snapshot option in your query to fix this problem.

(You could also use the hint option specifying the '_id' index.)

2
  • Yep, thanks man, that was it. I modified my query like this: MyModel.where({integer_array: {"$not" => {'$size' => 0}}}).hint("_id" => 1).each do |n| Jun 6, 2013 at 6:34
  • Thank you so much for this would have spent a lot of time trying to debug a similar problem without this answer
    – Leo Allen
    Feb 18, 2014 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.