111

Is there a way knowing (at coding time) which exceptions to expect when executing python code?

I end up catching the base Exception class 90% of the time since I don't know which exception type might be thrown (reading the documentation doesn't always help, since many times an exception can be propagated from the deep. And many times the documentation is not updated or correct).

Is there some kind of tool to check this (like by reading the Python code and libs)?

3
  • 3
    Bear in mind that in Python < 2.6, you can raise strings as well, not just BaseException subclasses. So if you're calling into library code that is out of your control, even except Exception is not sufficient, since it won't catch string exceptions. As others have pointed out, you're barking up the wrong tree here. Oct 19, 2009 at 22:42
  • I didn't know that. I thought except Exception:.. catches almost everything.
    – GabiMe
    Oct 19, 2009 at 22:59
  • 3
    except Exception works fine for catching string exceptions in Python 2.6 and later. Dec 12, 2009 at 2:30

8 Answers 8

25

I guess a solution could be only imprecise because of lack of static typing rules.

I'm not aware of some tool that checks exceptions, but you could come up with your own tool matching your needs (a good chance to play a little with static analysis).

As a first attempt, you could write a function that builds an AST, finds all Raise nodes, and then tries to figure out common patterns of raising exceptions (e. g. calling a constructor directly)

Let x be the following program:

x = '''\
if f(x):
    raise IOError(errno.ENOENT, 'not found')
else:
    e = g(x)
    raise e
'''

Build the AST using the compiler package:

tree = compiler.parse(x)

Then define a Raise visitor class:

class RaiseVisitor(object):
    def __init__(self):
        self.nodes = []
    def visitRaise(self, n):
        self.nodes.append(n)

And walk the AST collecting Raise nodes:

v = RaiseVisitor()
compiler.walk(tree, v)

>>> print v.nodes
[
    Raise(
        CallFunc(
            Name('IOError'),
            [Getattr(Name('errno'), 'ENOENT'), Const('not found')],
            None, None),
        None, None),
    Raise(Name('e'), None, None),
]

You may continue by resolving symbols using compiler symbol tables, analyzing data dependencies, etc. Or you may just deduce, that CallFunc(Name('IOError'), ...) "should definitely mean raising IOError", which is quite OK for quick practical results :)

9
  • Thanks for this interesting answer. Didn't understand though why should I look for something more other than all the raise nodes. Why should I "resolving symbols using compiler symbol tables, analyzing data dependencies"? Isn't the only way to raise exception is by raise() ?
    – GabiMe
    Oct 19, 2009 at 22:30
  • 1
    Given the v.nodes value above, you cannot actually say, what the thing is Name('IOError') or Name('e'). You don't know what value(s) those IOError and e can point to, as they are so-called free variables. Even if their binding context were known (here symbol tables come into play), you should perform some kind of data dependency analysis to infer their exact values (this should be hard in Python). Oct 19, 2009 at 22:41
  • As you are looking for a practical semi-automated solution, a list of ['IOError(errno.ENOENT, "not found")', 'e'] displayed to the user is just fine. But you cannot infer actual classes of values of variables represented by strings :) (sorry for reposting) Oct 19, 2009 at 22:46
  • 1
    Yes. This method, though clever, doesn't actually give you a complete coverage. Due to Python's dynamic nature, it is perfectly possible (though obviously a bad idea) for code you're calling into to do something like exc_class = raw_input(); exec "raise " + exc_class. The point is that this kind of static analysis is not truly possible in a dynamic language like Python. Oct 19, 2009 at 22:48
  • 8
    By the way, you can just find /path/to/library -name '*.py' | grep 'raise ' to get similar results :) Oct 19, 2009 at 22:54
25

You should only catch exceptions that you will handle.

Catching all exceptions by their concrete types is nonsense. You should catch specific exceptions you can and will handle. For other exceptions, you may write a generic catch that catches "base Exception", logs it (use str() function) and terminates your program (or does something else that's appropriate in a crashy situation).

If you really gonna handle all exceptions and are sure none of them are fatal (for example, if you're running the code in some kind of a sandboxed environment), then your approach of catching generic BaseException fits your aims.

You might be also interested in language exception reference, not a reference for the library you're using.

If the library reference is really poor and it doesn't re-throw its own exceptions when catching system ones, the only useful approach is to run tests (maybe add it to test suite, because if something is undocumented, it may change!). Delete a file crucial for your code and check what exception is being thrown. Supply too much data and check what error it yields.

You will have to run tests anyway, since, even if the method of getting the exceptions by source code existed, it wouldn't give you any idea how you should handle any of those. Maybe you should be showing error message "File needful.txt is not found!" when you catch IndexError? Only test can tell.

5
  • 40
    Sure, But how can one decide which exceptions he should handle if he doesn't know what might be thrown ??
    – GabiMe
    Oct 19, 2009 at 21:56
  • @bugspy.net , fixed my answer to reflect this matter
    – P Shved
    Oct 19, 2009 at 22:05
  • Maybe it's time for source code analyzer that can find out this? Shouldn't be too hard to develop i think
    – GabiMe
    Oct 19, 2009 at 22:10
  • @bugspy.net , I emboldened the clause why it might not be the time for it.
    – P Shved
    Oct 19, 2009 at 22:17
  • Sure you are right. However it may still be interesting - during development - to know which types of exceptions may occur.
    – hek2mgl
    Aug 17, 2015 at 22:41
17

The correct tool to solve this problem is unittests. If you are having exceptions raised by real code that the unittests do not raise, then you need more unittests.

Consider this

def f(duck):
    try:
        duck.quack()
    except ??? could be anything

duck can be any object

Obviously you can have an AttributeError if duck has no quack, a TypeError if duck has a quack but it is not callable. You have no idea what duck.quack() might raise though, maybe even a DuckError or something

Now supposing you have code like this

arr[i] = get_something_from_database()

If it raises an IndexError you don't know whether it has come from arr[i] or from deep inside the database function. usually it doesn't matter so much where the exception occurred, rather that something went wrong and what you wanted to happen didn't happen.

A handy technique is to catch and maybe reraise the exception like this

except Exception as e
    #inspect e, decide what to do
    raise
4
  • Why catch it at all if you are going to "reraise" it? Oct 19, 2009 at 22:25
  • You don't have to reraise it, that is what the comment was supposed to indicate. Oct 19, 2009 at 22:28
  • 3
    You may also choose to log the exception somewhere and then reraise Oct 19, 2009 at 22:40
  • 13
    I don't think that writing unit tests is the answer. The question is "how can I find out which exceptions to expect" and writing unit tests will not help you find this out. In fact, in order to write the unit test you already have to know which exceptions to expect so in order to write a correct unit test you have to anser the original question as well. Jun 28, 2019 at 21:52
9

Noone explained so far, why you can't have a full, 100% correct list of exceptions, so I thought it's worth commenting on. One of the reasons is a first-class function. Let's say that you have a function like this:

def apl(f,arg):
   return f(arg)

Now apl can raise any exception that f raises. While there are not many functions like that in the core library, anything that uses list comprehension with custom filters, map, reduce, etc. are affected.

The documentation and the source analysers are the only "serious" sources of information here. Just keep in mind what they cannot do.

5

I ran into this when using socket, I wanted to find out all the error conditions I would run in to (so rather than trying to create errors and figure out what socket does I just wanted a concise list). Ultimately I ended up grep'ing "/usr/lib64/python2.4/test/test_socket.py" for "raise":

$ grep raise test_socket.py
Any exceptions raised by the clients during their tests
        raise TypeError, "test_func must be a callable function"
    raise NotImplementedError, "clientSetUp must be implemented."
    def raise_error(*args, **kwargs):
        raise socket.error
    def raise_herror(*args, **kwargs):
        raise socket.herror
    def raise_gaierror(*args, **kwargs):
        raise socket.gaierror
    self.failUnlessRaises(socket.error, raise_error,
    self.failUnlessRaises(socket.error, raise_herror,
    self.failUnlessRaises(socket.error, raise_gaierror,
        raise socket.error
    # Check that setting it to an invalid value raises ValueError
    # Check that setting it to an invalid type raises TypeError
    def raise_timeout(*args, **kwargs):
    self.failUnlessRaises(socket.timeout, raise_timeout,
    def raise_timeout(*args, **kwargs):
    self.failUnlessRaises(socket.timeout, raise_timeout,

Which is a pretty concise list of errors. Now of course this only works on a case by case basis and depends on the tests being accurate (which they usually are). Otherwise you need to pretty much catch all exceptions, log them and dissect them and figure out how to handle them (which with unit testing wouldn't be to difficult).

2
  • 4
    This strengthen my argument, that exception handling in Python is very problematic, if we need to use grep or source analyzers to deal with something so basic (which for example in java existed from day one. Sometimes verbosity is a good thing. Java is verbose but at least there are no nasty surprises)
    – GabiMe
    Oct 20, 2009 at 10:56
  • @GabiMe, It's not like this ability (or static typing in general) is a silver bullet to prevent all bugs. Java is full of nasty surprises. That's why eclipse regularly crashes. Aug 16, 2013 at 20:29
2

There are two ways that I found informative. The first one, run the code in iPython, which will display the exception type.

n = 2
str = 'me '
str + 2
TypeError: unsupported operand type(s) for +: 'int' and 'str'

In the second way we settle for catching too much and improve on it over time. Include a try expression in your code and catch except Exception as err. Print sufficient data to know what exception was thrown. As exceptions are thrown improve your code by adding a more precise except clause. When you feel that you have caught all relevant exceptions remove the all inclusive one. A good thing to do anyway because it swallows programming errors.

try:
   so something
except Exception as err:
   print "Some message"
   print err.__class__
   print err
   exit(1)
1

normally, you'd need to catch exception only around a few lines of code. You wouldn't want to put your whole main function into the try except clause. for every few line you always should now (or be able easily to check) what kind of exception might be raised.

docs have an exhaustive list of built-in exceptions. don't try to except those exception that you're not expecting, they might be handled/expected in the calling code.

edit: what might be thrown depends on obviously on what you're doing! accessing random element of a sequence: IndexError, random element of a dict: KeyError, etc.

Just try to run those few lines in IDLE and cause an exception. But unittest would be a better solution, naturally.

2
  • 1
    This not answering my simple question. I don't ask on how to design my exception handling, or when or how to catch. I ask how to find out what might be thrown
    – GabiMe
    Oct 19, 2009 at 21:59
  • 1
    @bugspy.net: It is impossible to do what you ask, and this is a perfectly valid workaround. Oct 19, 2009 at 22:44
0

This is a copy and pasted answer I wrote for How to list all exceptions a function could raise in Python 3?, I hope that is allowed.

I needed to do something similar and found this post. I decided I would write a little library to help.

Say hello to Deep-AST. It's very early alpha but it is pip installable. It has all of the limitations mentioned in this post and some additional ones but its already off to a really good start.

For example when parsing HTTPConnection.getresponse() from http.client it parses 24489 AST Nodes. It finds 181 total raised Exceptions (this includes duplicates) and 8 unique Exceptions were raised. A working code example.

The biggest flaw is this it currently does work with a bare raise:

def foo():

    try:
        bar()
    except TypeError:
        raise

But I think this will be easy to solve and I plan on fixing it.

The library can handle more than just figuring out exceptions, what about listing all Parent classes? It can handle that too!

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.