43

I am trying to iterate through this loop:

for doc in coll.find()

I get the following error at the 100,000th plus record.

File "build\bdist.win32\egg\pymongo\cursor.py", line 703, in next
File "build\bdist.win32\egg\pymongo\cursor.py", line 679, in _refresh
File "build\bdist.win32\egg\pymongo\cursor.py", line 628, in __send_message
File "build\bdist.win32\egg\pymongo\helpers.py", line 95, in _unpack_response
pymongo.errors.OperationFailure: cursor id '1236484850793' not valid at server

what does this error mean?

2

5 Answers 5

38

Maybe your cursor timed out on the server. To see if this is the problem, try to set timeout=False`:

for doc in coll.find(timeout=False)

See http://api.mongodb.org/python/1.6/api/pymongo/collection.html#pymongo.collection.Collection.find

If it was a timeout problem one possible solution is to set the batch_size (s. other answers).

7
  • 2
    The FAQ suggests you have this correct: api.mongodb.org/python/current/… Apr 24, 2012 at 13:10
  • im at the 50,000th record. waiting to see if im thru :)
    – codious
    Apr 24, 2012 at 13:14
  • 1
    Please note that this consumes memory on the server until you explicitly close it. See mongodb.org/display/DOCS/… Nov 7, 2012 at 16:07
  • @CraigYounkins correct me if I'm wrong, but with pymongo, I believe a cursor object is closed when it is garbage collected, so this isn't such a problem unless there is something like a network failure (so pymongo can't tell the server to close it).
    – drevicko
    Mar 20, 2015 at 11:22
  • 1
    I just looked it up in the pymongo source: yes, it does get closed when the cursor is garbage collected. Network failures and broken garbage collection (e.g.: circular refs in python destructors) could stop that from working though.
    – drevicko
    Mar 23, 2015 at 4:31
35
+150
  • Setting the timeout=False is dangerous and should never be used, because the connection to the cursor can remain open for unlimited time, which will affect system performance. The docs specifically reference the need to manually close the cursor.
  • Setting the batch_size to a small number will work, but creates a big latency issue, because we need to access the DB more often than needed.
    For example:
    5M docs with a small batch will take hours to retrieve the same data that a default batch_size returns in several minutes.

In my solution it is mandatory to use sort on the cursor:

done = False
skip = 0
while not done:
    cursor = coll.find()
    cursor.sort( indexed_parameter ) # recommended to use time or other sequential parameter.
    cursor.skip( skip )
    try:
        for doc in cursor:
            skip += 1
            do_something()
        done = True
    except pymongo.errors.OperationFailure, e:
        msg = e.message
        if not (msg.startswith("cursor id") and msg.endswith("not valid at server")):
            raise
4
  • This is a nice solution, although if you have a few million entries and no "time or other sequential parameter" it's impractical. Can't believe there is no off the self solution. Jan 30, 2014 at 8:31
  • 1
    Just to make it clear. This solution (either the batch one) do not ensure iterate all the documents just once. Some document could be yield more than once or skipped if the database get updated between queries. For statistical proposes this is usually not a problem however if you need exactitude this may be a problem in some cases. Jun 23, 2015 at 10:12
  • "timeout=False is dangerous and should never be used" this is untrue - if I have a script which iterates over collection elements, one by one and on each I spent 5s-15min period of time, then I should be using this option. There is no risk of cursor remaining afterwards since it will be destroyed along with the collection at the end of script. Jan 6, 2020 at 12:10
  • @MaciejUrbański This was true according to Mongo 2.4/2.6 docs. This parameter was changed, even by name, and probably by functionality. The question doesn't have a tag to the Mongo version, so this is a bit confusing. Anyway, keeping cursors open is dangerous due to potentially high memory usage, if you open many cursors and forget to close them. Consider this option as an advanced user feature, which is usually not something you want to do - although there are always exceptions. Thanks for you feedback :)
    – Oran
    Jan 9, 2020 at 15:23
26

Setting timeout=False is a very bad practice. A better way to get rid of the cursor id timeout exception is to estimate how many documents your loop can process within 10 minutes, and come up with an conservative batch size. This way, the MongoDB client (in this case, PyMongo) will have to query the server once in a while whenever the documents in the previous batch were used up. This will keep the cursor active on the server, and you will still be covered by the 10-minute timeout protection.

Here is how you set batch size for a cursor:

for doc in coll.find().batch_size(30):
    do_time_consuming_things()
1
  • I agree, this sounds like the better solution
    – Tom Swifty
    Sep 25, 2013 at 12:54
0

you should choose a low value of batch_size to fix the issue:

col.find({}).batch_size(10)

see the following answer

-1

You can also force evaluation by using:

for doc in list(coll.find())
1
  • @peterh The question is about fixing the cursor timeout issue, not about explaining how cursors and batches work. I agree a more detailed explanation would be great, but this answer is still valid, as converting the cursor to list will force it to retrieve all the batches and close, most likely before the 10 minutes default expiration time.
    – Danziger
    May 30, 2017 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.