Python list comprehensions are nice, but near impossible to debug. You guys have any good tips / tools for debugging them?
8 Answers
I use a function that just prints and returns a value at the same time:
from pprint import pprint
def debug(msg, item):
print('\n' + msg + ':')
pprint(item)
return item
It's very handy for debugging any part of a list/dict comprehension:
new_lines = [
debug('CUR UPDATED LINE', change(line))
for line
in debug('ALL LINES', get_lines_from_file(filename))
if debug('CUR LINE EMPTY?', not_empty(line))
]
-
I love this answer! Shame to see it at the bottom, it's the natural equivalent of print-debugging in functional programming. Commented Aug 8, 2017 at 0:06
-
@Rotareti, have you considered to edit my answer from 2010 that is exactly same approach? ;)– onyCommented Nov 24, 2019 at 20:12
-
@ony For some reason I hadn't noticed that you used a similar approach when I was browsing through the answers. I know that sucks, and I'm sorry. :-/– RotaretiCommented Nov 24, 2019 at 22:43
It depends on the list comprehension. You can move a part of the code to another function. This should be a clean solution which is more easy to debug.
Example:
[1.0 / i for i in [0, 2, 5, 10]]
Can be divided into
[f(i) for i in [0, 2, 5, 10]]
and a function
def f(i):
return 1.0 / i
When you do the debugging you will find out it will crash because of a “division-by-zero” error at f
for the value of i = 0
.
-
By the way you have a spelling error ("devision" instead of "division") that I can't fix for you because it's only a two-character fix and SE won't let me make a fix that's less than 6 characters.– bobCommented Oct 3, 2019 at 14:12
-
No problem. Hopefully SE will figure out a way to let us make those types of fixes ourselves.– bobCommented Oct 3, 2019 at 16:38
-
This works but is awkward: need to define an out-of-line separate function. Well that's python.. Commented Mar 5, 2021 at 6:16
If it's complicated enough that it's not obvious at first glance, unpack it into multiple steps and/or for loops. It's clearly too complicated, and making it more explicit is the easiest way to go about debugging it. Added bonus: you can now step through with the debugger or add print statements!
-
1+1: pythonic usage of list comprehensions are when the code actually gets clearer and easier to read– noskloCommented Mar 29, 2010 at 5:15
-
14
-
1Sometimes surrendering is the correct course of action! If you can't read it, it's clearly too complicated for a one-liner. Commented Jul 8, 2015 at 23:11
-
1Sometimes the list comprehension logic isn't complicated, but there's a bug somewhere in a method or function that it calls.– bobCommented Oct 4, 2018 at 17:33
-
1My problem with this argument is that the reason I prefer list comprehensions over an equally-complex for-loop is that list comprehensions are declarative and I'm thus less likely to mess things up, rather than building the list in a for-loop with list.append calls. I honestly don't see why Python can't add debugging support for list comprehensions. As it stands today, basically I write a comprehension and as long as it never gives me problems, great! Once it does, it's back to the old for-loop. It shouldn't be this way.– bobCommented Oct 3, 2019 at 14:09
In Haskell I using something similar to:
def trcPV(prompt, value):
print ("%s%s" % (prompt, str(value)))
return value
xs = trcPV("xs=", [x for x in range(0,100) if trcPV("check=",(trcPV("x=",x) % 15) in [0,3,5])])
-
@bob, ermm... first 3 lines is a proposed function similar to
trace
in Haskell, that helps to trace which values are being calculated while passing value unmodified further where function placed in. If you previously hadx+y
you can writetrcPV("x=", x) + trcPV("y=", y)
.– onyCommented Nov 17, 2019 at 22:57
tip: Use list comprehension for simple tasks (1 or 2 levels). Otherwise, making it explicit is better for readability.
-
1But what do you do if you have a bug in a comprehension of this type?– bobCommented Oct 3, 2019 at 14:10
Use a debugger like pdb
to walk through or break the list comprehension into a full for loop.
-
2Could you elaborate on how you use pdb to walk through? Have you successfully used an IDE debugger before (I've tried to use pdb before and have given up in frustration each time)?– bobCommented Oct 3, 2019 at 14:15
-
pdb runs list comprehensions in their own context in the debugger, It will say local variables are not defined and generally just call over. Commented Mar 31, 2022 at 12:29
Haskell list comprehensions at least can be (and that is what compilers do) rewritten in terms of map, concat and filter.
So this Haskell example:
[ x*x | x<-[1..25], even x]
Works out as:
map (\x-> x*x) (filter (even) [1..25])
I expect similar identities would continue to hold for Python, so similar decomposition should yield equivalent code in Python as well. The equivalent code should prove easier to debug (and run about as efficiently).
-
It's true that you can debug compositions of map and filter in Python as long as you use declared functions. The problem is that list comprehensions are more readable, and this answer, while true, is basically saying "don't use comprehensions". The goal is to learn how to use comprehensions and debug them.– bobCommented Oct 3, 2019 at 14:14
-
In python3 every map/filter/reduce is not only tacked on backwards but also requires an extra
list(*)
since python thinks everything should be a generator by default. Completely unreadable beyond a single level of nesting. Commented Mar 5, 2021 at 6:18
Building on Elmex80s very nice response in https://stackoverflow.com/a/39350403/5339264, using a debug function can also help with TypeError: unsupported operand type(s) for +: 'method' and 'str'
or similar errors in a list comprehension.
A classic
def debug(i):
print(f"i: {i}, {str(type(i))}")
in a list comprehension like
[debug(item) for item in list]
can be very useful to unravel what item in the list is causing the error.
map/filter/reduce
are unusable after one or two levels of nesting (and lousy to read even at one).. I have resorted to libraries that do what they can to allow piping of collections processing (map/filter/reduce and friends) through a chain of operations that behind the scenes handle remembering the collections data.fluentpy
is one. That approach is absolutely non-pythonic and yet far superior to being limited to 2 levels of a for comprehension nesting