24

Python list comprehensions are nice, but near impossible to debug. You guys have any good tips / tools for debugging them?

2
  • It looks like so far the answers are: use a special-case of print debugging or don't use comprehensions at all. Are there any tools out there that support this without having to modify your code? It seems strange to me that this is an intractable problem.
    – bob
    Commented Oct 3, 2019 at 14:17
  • There's no builtin solution to this problem. In addition to the comprehensions weakness python's built-in map/filter/reduce are unusable after one or two levels of nesting (and lousy to read even at one).. I have resorted to libraries that do what they can to allow piping of collections processing (map/filter/reduce and friends) through a chain of operations that behind the scenes handle remembering the collections data. fluentpy is one. That approach is absolutely non-pythonic and yet far superior to being limited to 2 levels of a for comprehension nesting Commented Mar 5, 2021 at 6:11

8 Answers 8

22

I use a function that just prints and returns a value at the same time:

from pprint import pprint

def debug(msg, item):
    print('\n' + msg + ':')
    pprint(item)
    return item

It's very handy for debugging any part of a list/dict comprehension:

new_lines = [
    debug('CUR UPDATED LINE', change(line))
    for line
    in debug('ALL LINES', get_lines_from_file(filename))
    if debug('CUR LINE EMPTY?', not_empty(line))
    ]
3
  • I love this answer! Shame to see it at the bottom, it's the natural equivalent of print-debugging in functional programming. Commented Aug 8, 2017 at 0:06
  • @Rotareti, have you considered to edit my answer from 2010 that is exactly same approach? ;)
    – ony
    Commented Nov 24, 2019 at 20:12
  • @ony For some reason I hadn't noticed that you used a similar approach when I was browsing through the answers. I know that sucks, and I'm sorry. :-/
    – Rotareti
    Commented Nov 24, 2019 at 22:43
6

It depends on the list comprehension. You can move a part of the code to another function. This should be a clean solution which is more easy to debug.

Example:

[1.0 / i for i in [0, 2, 5, 10]]

Can be divided into

[f(i) for i in [0, 2, 5, 10]] 

and a function

def f(i):         
    return 1.0 / i  

When you do the debugging you will find out it will crash because of a “division-by-zero” error at f for the value of i = 0.

3
  • By the way you have a spelling error ("devision" instead of "division") that I can't fix for you because it's only a two-character fix and SE won't let me make a fix that's less than 6 characters.
    – bob
    Commented Oct 3, 2019 at 14:12
  • No problem. Hopefully SE will figure out a way to let us make those types of fixes ourselves.
    – bob
    Commented Oct 3, 2019 at 16:38
  • This works but is awkward: need to define an out-of-line separate function. Well that's python.. Commented Mar 5, 2021 at 6:16
4

If it's complicated enough that it's not obvious at first glance, unpack it into multiple steps and/or for loops. It's clearly too complicated, and making it more explicit is the easiest way to go about debugging it. Added bonus: you can now step through with the debugger or add print statements!

5
  • 1
    +1: pythonic usage of list comprehensions are when the code actually gets clearer and easier to read
    – nosklo
    Commented Mar 29, 2010 at 5:15
  • 14
    That kind of feels like surrender
    – Carbon
    Commented Jul 3, 2015 at 2:40
  • 1
    Sometimes surrendering is the correct course of action! If you can't read it, it's clearly too complicated for a one-liner. Commented Jul 8, 2015 at 23:11
  • 1
    Sometimes the list comprehension logic isn't complicated, but there's a bug somewhere in a method or function that it calls.
    – bob
    Commented Oct 4, 2018 at 17:33
  • 1
    My problem with this argument is that the reason I prefer list comprehensions over an equally-complex for-loop is that list comprehensions are declarative and I'm thus less likely to mess things up, rather than building the list in a for-loop with list.append calls. I honestly don't see why Python can't add debugging support for list comprehensions. As it stands today, basically I write a comprehension and as long as it never gives me problems, great! Once it does, it's back to the old for-loop. It shouldn't be this way.
    – bob
    Commented Oct 3, 2019 at 14:09
2

In Haskell I using something similar to:

def trcPV(prompt, value):
    print ("%s%s" % (prompt, str(value)))
    return value

xs = trcPV("xs=", [x for x in range(0,100) if trcPV("check=",(trcPV("x=",x) % 15) in [0,3,5])])
1
  • @bob, ermm... first 3 lines is a proposed function similar to trace in Haskell, that helps to trace which values are being calculated while passing value unmodified further where function placed in. If you previously had x+y you can write trcPV("x=", x) + trcPV("y=", y).
    – ony
    Commented Nov 17, 2019 at 22:57
1

tip: Use list comprehension for simple tasks (1 or 2 levels). Otherwise, making it explicit is better for readability.

1
  • 1
    But what do you do if you have a bug in a comprehension of this type?
    – bob
    Commented Oct 3, 2019 at 14:10
1

Use a debugger like pdb to walk through or break the list comprehension into a full for loop.

2
  • 2
    Could you elaborate on how you use pdb to walk through? Have you successfully used an IDE debugger before (I've tried to use pdb before and have given up in frustration each time)?
    – bob
    Commented Oct 3, 2019 at 14:15
  • pdb runs list comprehensions in their own context in the debugger, It will say local variables are not defined and generally just call over. Commented Mar 31, 2022 at 12:29
0

Haskell list comprehensions at least can be (and that is what compilers do) rewritten in terms of map, concat and filter.

So this Haskell example:

[ x*x | x<-[1..25], even x]

Works out as:

map (\x-> x*x) (filter (even) [1..25])

I expect similar identities would continue to hold for Python, so similar decomposition should yield equivalent code in Python as well. The equivalent code should prove easier to debug (and run about as efficiently).

2
  • It's true that you can debug compositions of map and filter in Python as long as you use declared functions. The problem is that list comprehensions are more readable, and this answer, while true, is basically saying "don't use comprehensions". The goal is to learn how to use comprehensions and debug them.
    – bob
    Commented Oct 3, 2019 at 14:14
  • In python3 every map/filter/reduce is not only tacked on backwards but also requires an extra list(*) since python thinks everything should be a generator by default. Completely unreadable beyond a single level of nesting. Commented Mar 5, 2021 at 6:18
0

Building on Elmex80s very nice response in https://stackoverflow.com/a/39350403/5339264, using a debug function can also help with TypeError: unsupported operand type(s) for +: 'method' and 'str' or similar errors in a list comprehension.

A classic

def debug(i):
    print(f"i: {i}, {str(type(i))}")

in a list comprehension like

[debug(item) for item in list]

can be very useful to unravel what item in the list is causing the error.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.