13

Quickie here that needs more domain expertise on pymongo than I have right now:

Are the "right" parts of the pymongo driver written in python for me to call gevent monkey_patch() and successfully alter pymongo's blocking behavior on r/w within gevent "asynchronous" greenlets?

If this will require a little more leg work on gevent and pymongo -- but it is feasible -- I would be more than willing to put in the time as long as i can get a little guidance over irc.

Thanks!

Note: At small scale mongo writes are not a big problem because we are just queuing a write "request" before unblocking. BUT talking to fiorix about his twisted async mongo driver (https://github.com/fiorix/mongo-async-python-driver), even mongo's quick write (requests) can cause problems in asyncronous applications at scale. (And of course, non-blocking reads could cause problems from the start!)

2 Answers 2

19

I have used PyMongo with Gevent and here are a few things you need to watch out for:

  1. Instantiate only one pymongo.Connection object, preferrably as a global or module-level variable. This is important because Connection has within itself a pool!
  2. Monkey patch everything, or at least BOTH socket and threading. Due to the use of thread locals in Connection, patching socket alone is not enough.
  3. Remember to call end_request to return the connection to the pool.

The answer to your question is go ahead, PyMongo works just fine with Gevent.

3
  • thanks Bernie Hackett on mongodb google group for this update: there is still the problem that mongo has an upper limit on its connection pool much lower than the number of greenlets that most app's will be spawning if they are using mongo in the first place (1000's of greenlets). Some of the newest mongodb patches here github.com/mgood/mongo-python-driver are in an attempt to make gevent play nicely with mongo's thread pooling. Here is another example of a patch that might work: code.activestate.com/recipes/….
    – egbutter
    Aug 26, 2011 at 17:11
  • Do you have, or know of, any examples that show PyMongo being used with Gevent? Feb 3, 2012 at 1:29
  • And here is yet another example of a patch that might work: gist.github.com/1184264
    – kkurian
    Feb 4, 2012 at 6:36
2

On initial inspection it doesn't appear to do any socket operations in the c code so it should be fine (blocking ops should just block the green thread).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.