There are several ways to iterate over a result set. What are the tradeoff of each?
3 Answers
The canonical way is to use the built-in cursor iterator.
curs.execute('select * from people')
for row in curs:
print row
You can use fetchall()
to get all rows at once.
for row in curs.fetchall():
print row
It can be convenient to use this to create a Python list containing the values returned:
curs.execute('select first_name from people')
names = [row[0] for row in curs.fetchall()]
This can be useful for smaller result sets, but can have bad side effects if the result set is large.
You have to wait for the entire result set to be returned to your client process.
You may eat up a lot of memory in your client to hold the built-up list.
It may take a while for Python to construct and deconstruct the list which you are going to immediately discard anyways.
If you know there's a single row being returned in the result set you can call fetchone()
to get the single row.
curs.execute('select max(x) from t')
maxValue = curs.fetchone()[0]
Finally, you can loop over the result set fetching one row at a time. In general, there's no particular advantage in doing this over using the iterator.
row = curs.fetchone()
while row:
print row
row = curs.fetchone()
-
1about the second method, what if you use a SScursor ? will it stil eat up a lot of memory?– SylvainNov 27, 2009 at 10:50
-
I think SScursor is for MySQL. But anything that has a fetchall() will probably have the same memory usage, as it returns a list of all the rows returned. Nov 27, 2009 at 20:25
My preferred way is the cursor iterator, but setting first the arraysize property of the cursor.
curs.execute('select * from people')
curs.arraysize = 256
for row in curs:
print row
In this example, cx_Oracle will fetch rows from Oracle 256 rows at a time, reducing the number of network round trips that need to be performed
-
2In my tests (on a database connected by LAN) this actually seemed to give identical (even slower, in a couple iterations) speed as compared to doing 'fetchone()' repeatedly. I was doing it with about 12000 entries... Very odd!– DKGasserJul 1, 2011 at 16:01
-
The only way that I know of, and I'm in no way an Oracle expert, that this would be the case is if your query is returning character large object (CLOB) or binary large object (BLOB) types. AFAI Understand it, reading these objects requires another network round trip to the db server for each record; meaning that with fetchmany you actually get the worst of both worlds. Nov 5, 2015 at 20:31
-
1For cx_Oracle, connecting to a 12c database with standard column types (no clobs etc) I get a speedup but only if I set arraysize before executing the query. Precise numbers will obviously be massively context dependent but to give an idea of the order of magnitude changes, my query (returning 5 columns) with arraysize=50 (default) gives 3.75us per row. Decreasing arraysize to 1 gives 70us. Increasing arraysize to 1000 gives 800ns– FredLJul 6, 2016 at 9:39
-
1@FredL I'm seeing something similar. It's making a difference only when set before the
execute
call.– AmanDec 11, 2017 at 3:15
There's also the way psyco-pg
seems to do it... From what I gather, it seems to create dictionary-like row-proxies to map key lookup into the memory block returned by the query. In that case, fetching the whole answer and working with a similar proxy-factory over the rows seems like useful idea. Come to think of it though, it feels more like Lua than Python.
Also, this should be applicable to all PEP-249 DBAPI2.0 interfaces, not just Oracle, or did you mean just fastest using Oracle?