4

I've been doing some reading on using SQLAlchemy's ORM in the context of a Twisted application. It's a lot of information to digest, so I'm having a bit of trouble putting all the pieces together. So far, I've gathered the following absolute truths:

  1. One session implies one thread. Always.
  2. scoped_session, by default, provides us with a way of constraining sessions to a given thread. In other words, I am sure that by using scoped_session, I will not pass sessions to other threads (unless I do so explicitly, which I won't).

I also gathered that there are some issues relating to lazy/eager-loading and that one possible approach is to dissociate ORM objects from a session and reattach them to another session when changing threads. I'm quite fuzzy on the details, but I also concluded that scoped_session renders many of these points moot.

My first question is whether or not I am severely mistaken in my above conclusions.

Beyond that, I've crafted this approach, which I hope is satisfactory.

I begin by creating a scoped_session object...

Session = scoped_session(sessionmaker(bind=_my_engine))

... which I will then use from a context manager, in order to handle exceptions and clean-up gracefully:

@contextmanager
def transaction_context():
    session = Session()
    try:
        yield session
        session.commit()
    except:
        session.rollback()
        raise
    finally:
        session.remove()  # dispose of the session

Now all I need to do is to use the above context manager in a function that is deferred to a separate thread. I've thrown together a decorator to make things a bit prettier:

def threaded(fn):
    @wraps(fn)  # functools.wraps
    def wrapper(*args, **kwargs):
        return deferToThread(fn, *args, **kwargs)  # t.i.threads.deferToThread
    return wrapper

Here is an example of how I intend to use the whole shebang. Below is a function that performs a DB lookup using the SQLAlchemy ORM:

@threaded
def get_some_attributes(group):
    with transaction_context() as session:
        return session.query(Attribute).filter(Attribute.group == group)

My second question is whether or not this approach is viable.

  • Am I making any fundamentally flawed assumptions?
  • Are there any caveats?
  • Is there a better way?

Edit: Here is a related question concerning the unexpected error in my context manager.

9
  • Well, I guess the foremost question that everyone will have is: does it work the way that you coded it?
    – bitcycle
    Jan 12, 2014 at 17:07
  • @bitcycle, Incidentally (and surprisingly), no... it doesn't work. I'm getting a AttributeError in my context manager -- apparently Session has no attribute remove. This is rather surprising -- a Jan 12, 2014 at 18:04
  • 1
    This looks mostly right (except it should be Session.remove()). One thing to keep in mind is that some database drivers, such as sqlite3, don't allow their connections to be passed between threads. I'm fairly sure scoped_session objects handle this appropriately (it's been a while since I used SQLAlchemy). Jan 13, 2014 at 3:56
  • 1
    As @Glyph mentioned, Alchimia already does some of this. It works for the most part (although it's still fairly incomplete) but it doesn't yet have the thread-pinning necessary to make sqlite and friends happy. Contributions are welcome and it's probably a better starting point than doing it from scratch.
    – jerith
    Jan 14, 2014 at 8:02
  • 1
    I am pretty sure Alchimia does not wrap the ORM. (Perhaps "yet", perhaps few people actually like the ORM?)
    – Glyph
    Jan 15, 2014 at 1:47

1 Answer 1

3

Right now I work on this exact problem, and I think I found a solution.

Indeed, you must defer all database access functions to a thread. But in your solution, you remove the session after querying the database, so all your results ORM objects will be detached and you wont have access to their fields.

You can't use scoped_session because in Twisted we have only one MainThread (except with things that work in deferToThread). We can, however, use scoped_sesssion with scopefunc.

In Twisted there is a great thing known as ContextTracker:

provides a way to pass arbitrary key/value data up and down a call stack without passing them as parameters to the functions on that call stack.

In my twisted web app in method render_GET I set a uuid parameter:

call = context.call({"uuid": str(uuid.uuid4())}, self._render, request)

and then I call the _render method to do the actual work (work with db, render html, etc).

I create the scoped_session like this:

scopefunc = functools.partial(context.get, "uuid")
Session = scoped_session(session_factory, scopefunc=scopefunc)

Now within any function calls of _render I can get session with:

Session()

and at the end of _render I have to do Session.remove() to remove the session.

It worksa with my webapp and I think can work for other tasks.

This is completely standalone example, show how all it work together.

from twisted.internet import reactor, threads
from twisted.web.resource import Resource
from twisted.web.server import Site, NOT_DONE_YET
from twisted.python import context
from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.orm import sessionmaker, scoped_session
from sqlalchemy.ext.declarative import declarative_base
import uuid
import functools

engine = create_engine(
    'sqlite:///test.sql',
    connect_args={'check_same_thread': False},
    echo=False)

session_factory = sessionmaker(bind=engine)
scopefunc = functools.partial(context.get, "uuid")
Session = scoped_session(session_factory, scopefunc=scopefunc)
Base = declarative_base()


class User(Base):
    __tablename__ = 'users'
    id = Column(Integer, primary_key=True)
    name = Column(String)

Base.metadata.create_all(bind=engine)


class TestPage(Resource):
    isLeaf = True

    def render_GET(self, request):
        context.call({"uuid": str(uuid.uuid4())}, self._render, request)
        return NOT_DONE_YET

    def render_POST(self, request):
        return self.render_GET(request)

    def work_with_db(self):
        user = User(name="TestUser")
        Session.add(user)
        Session.commit()
        return user

    def _render(self, request):
        print "session: ", id(Session())
        d = threads.deferToThread(self.work_with_db)

        def success(result):
            html = "added user with name - %s" % result.name
            request.write(html.encode('UTF-8'))
            request.finish()
            Session.remove()
        call = functools.partial(context.call, {"uuid": scopefunc()}, success)
        d.addBoth(call)
        return d

if __name__ == "__main__":
    reactor.listenTCP(8888, Site(TestPage()))
    reactor.run()

I print out id of session, and you can see that its different for each request. If you remove scopefunc from scoped_session constructor and do two simultaneous request(insert time.sleep to work_with_db), you will get one common session for this two requests.

The scoped_session object by default uses threading.local() as storage, so that a single Session is maintained for all who call upon the scoped_session registry, but only within the scope of a single thread

a problem here that in twisted we have only one thread for all requests. Thats why we have to create own scopefunc, that will show the difference between requests.

An other problem, that twisted didnt pass context to callbacks and we have to wrap callback and send current context to it.

call = functools.partial(context.call, {"uuid": scopefunc()}, success)

Still I dont know how to make it work with defer.inLineCallback, that I use everywhere in my code.

3
  • Thank you for your answer! Would you mind posting a more detailed example that explicitly shows how ContextTracker, _render and scoped_session all fit together? I'm having trouble seeing the big picture. Why is the scopefunc necessary with the scoped_session? Thank you! Feb 13, 2014 at 16:32
  • @blz add example and explanation of scopefunc using.
    – aborilov
    Feb 14, 2014 at 7:56
  • 1
    @aborilov Thanks! Nice solution you got here... I wonder if this could be done a bit more implicitly with a decorator
    – Kevin R.
    Nov 13, 2016 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.