31

I can't figure out what kind of object a sqlalchemy query returns.

entries = session.query(Foo.id, Foo.date).all()

The type of each object in entries seems to be sqlalchemy.util._collections.result, but a quick from sqlalchemy.util._collections import result in a python interpreter raises an ImportError.

What I'm ultimately trying to do is to type hint this function:

def my_super_function(session: Session) -> ???:
    entries = session.query(Foo.id, Foo.date).all()
    return entries

What should I put in place of ???? mypy (in this case) seems to be fine with List[Tuple[int, str]] because yes indeed I can access my entries like if they were tuples, but i can also access them with entry.date, for example.

2 Answers 2

35

I too found it curious that the class couldn't be imported. The answer is pretty long as I've walked you through how I've worked it out, bear with me.

Query.all() calls list() on the Query object itself:

def all(self):
    """Return the results represented by this ``Query`` as a list.
    This results in an execution of the underlying query.
    """
    return list(self)

... where list will be iterating over the object, so Query.__iter__():

def __iter__(self):
    context = self._compile_context()
    context.statement.use_labels = True
    if self._autoflush and not self._populate_existing:
        self.session._autoflush()
    return self._execute_and_instances(context)

... returns the result of Query._execute_and_instances() method:

def _execute_and_instances(self, querycontext):
    conn = self._get_bind_args(
        querycontext, self._connection_from_session, close_with_result=True
    )

    result = conn.execute(querycontext.statement, self._params)
    return loading.instances(querycontext.query, result, querycontext)

Which executes the query and returns the result of sqlalchemy.loading.instances() function. In that function there is this line which applies to non-single-entity queries:

keyed_tuple = util.lightweight_named_tuple("result", labels)

... and if I stick a print(keyed_tuple) in after that line it prints <class 'sqlalchemy.util._collections.result'>, which is the type that you mention above. So whatever that object is, it's coming from the sqlalchemy.util._collections.lightweight_named_tuple() function:

def lightweight_named_tuple(name, fields):
    hash_ = (name,) + tuple(fields)
    tp_cls = _lw_tuples.get(hash_)
    if tp_cls:
        return tp_cls

    tp_cls = type(
        name,
        (_LW,),
        dict(
            [
                (field, _property_getters[idx])
                for idx, field in enumerate(fields)
                if field is not None
            ]
            + [("__slots__", ())]
        ),
    )

    tp_cls._real_fields = fields
    tp_cls._fields = tuple([f for f in fields if f is not None])

    _lw_tuples[hash_] = tp_cls
    return tp_cls

So the key part is this statement:

tp_cls = type(
    name,
    (_LW,),
    dict(
        [
            (field, _property_getters[idx])
            for idx, field in enumerate(fields)
            if field is not None
        ]
        + [("__slots__", ())]
    ),
)

... which calls the built in type() class which according to the docs:

With three arguments, return a new type object. This is essentially a dynamic form of the class statement.

And this is why you cannot import the class sqlalchemy.util._collections.result - because the class is only constructed at query time. I'd say that the reason for this is that the column names (i.e. the named tuple attributes) aren't known until the query is executed).

From python docs the signature for type is: type(name, bases, dict) where:

The name string is the class name and becomes the __name__ attribute; the bases tuple itemizes the base classes and becomes the __bases__ attribute; and the dict dictionary is the namespace containing definitions for class body and is copied to a standard dictionary to become the __dict__ attribute.

As you can see, the bases argument passed to type() in lightweight_named_tuple() is (_LW,). So any of the dynamically created named tuple types inherit from sqlalchemy.util._collections._LW, which is a class that you can import:

from sqlalchemy.util._collections import _LW

entries = session.query(Foo.id, Foo.date).all()
for entry in entries:
    assert isinstance(entry, _LW)  # True

... so I'm not sure whether it's good form to type your function to an internal class with the leading underscore, but _LW inherits from sqlalchemy.util._collections.AbstractKeyedTuple, which itself inherits from tuple. That's why your current typing of List[Tuple[int, str]] works, because it is a list of tuples. So take your pick, _LW, AbstractKeyedTuple, tuple would all be correct representations of what your function is returning.

4
  • 12
    This is why we need protocols. Commented Mar 27, 2019 at 6:50
  • List[Tuple[int, str]] may cause problems down the line, if one is then going to use the _asdict() method on each of the result's items: Cannot access member "_asdict" for type "Tuple[str, str]"
    – pmsoltani
    Commented Sep 1, 2020 at 13:22
  • 1
    This is a very good answer, however I want to add as of now _LW and AbstractKeyerTuple are no longer part of sqlalchemy.util._collections, might be changed or moved elsewhere
    – Matthias
    Commented Nov 21, 2022 at 14:41
  • 2
    @Matthias much has been done in sqlalchemy to make it more friendly for typing over the last year or two, so I suppose it’s not surprising to see some changes to the internals. There are going to be a lot of questions that need an update when 2.0 is released, I’ll try to find the time to update this answer.
    – SuperShoot
    Commented Nov 21, 2022 at 21:07
-2

Just print or log the type(entries) to see which type is used. No need to read through the module code.

Without checking it, the return might be the same as typical records you would get with cursor.fetchall(). Then, the type is just tuple - the Python built-in tuple. You do not need even need to import tuple to use tuple in the typing module.

Writing this without having tested it, yet, the main trick is anyway something else: use type(my_return_var) to see the type for the type hint.

Mind that a class with the module path must at first be imported.

Another example of how to use the "trick": type hint for a cursor object, taken from What is the right type hint for the returned sqlalchemy objects "cursor" and "cursor.fetchall()" records?. When the output of the type(...) is <class 'MySQLdb.cursors.Cursor'>, then you need

  • from MySQLdb.cursors import Cursor with Cursor as the type hint or
  • from MySQLdb import cursors with cursors.Cursor as the type hint or
  • import MySQLdb with MySQLdb.cursors.Cursor as the type hint.

This "trick" of getting the right type for typing is also at Type hints for SQLAlchemy engine and session objects.

2
  • The whole point of this Q&A was that the output of type(...) resulted in sqlalchemy.util._collections.result, which was a dynamically constructed named tuple and something not able to be imported like you suggest with this "trick".
    – SuperShoot
    Commented Aug 27, 2023 at 23:33
  • @SuperShoot I see that I misunderstood the question. This answer does not help since the question already knows the type sqlalchemy.util._collections.result by means of the output of type(...). I did not read that in the question as such. Shall I delete the question? Commented Mar 16 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.