I have this Python application that gets stuck from time to time and I can't find out where.
Is there any way to signal Python interpreter to show you the exact code that's running?
Some kind of on-the-fly stacktrace?
Related questions:
Join Stack Overflow to learn, share knowledge, and build your career.
|
I have this Python application that gets stuck from time to time and I can't find out where. Is there any way to signal Python interpreter to show you the exact code that's running? Some kind of on-the-fly stacktrace? Related questions: |
|||||
|
|
I have module I use for situations like this - where a process will be running for a long time but gets stuck sometimes for unknown and irreproducible reasons. Its a bit hacky, and only works on unix (requires signals):
To use, just call the listen() function at some point when your program starts up (You could even stick it in site.py to have all python programs use it), and let it run. At any point, send the process a SIGUSR1 signal, using kill, or in python:
This will cause the program to break to a python console at the point it is currently at, showing you the stack trace, and letting you manipulate the variables. Use control-d (EOF) to continue running (though note that you will probably interrupt any I/O etc at the point you signal, so it isn't fully non-intrusive. I've another script that does the same thing, except it communicates with the running process through a pipe (to allow for debugging backgrounded processes etc). Its a bit large to post here, but I've added it as a python cookbook recipe. |
|||||||||||||||||||||
|
|
I was looking for a while for a solution to debug my threads and I found it here thanks to haridsv. I use slightly simplified version employing the traceback.print_stack():
For my needs I also filter threads by name. |
||||
|
|
|
In case you need to do this with uWSGI, it has Python Tracebacker built-in and it's just matter of enabling it in the configuration (number is attached to the name for each worker):
Once you have done this, you can print backtrace simply by connecting to the socket:
|
|||
|
|
|
Getting a stack trace of an unprepared python program, running in a stock python without debugging symbols can be done with pyrasite. Worked like a charm for me in on Ubuntu Trusty:
(Hat tip to @Albert, whose answer contained a pointer to this, among other tools.) |
|||||||||||||||||
|
|
In Python 3, pdb will automatically install a signal handler the first time you use c(ont(inue)) in the debugger. Pressing Control-C afterwards will drop you right back in there. In Python 2, here's a one-liner which should work even in relatively old versions (tested in 2.7 but I checked Python source back to 2.4 and it looked okay):
pdb is worth learning if you spend any amount of time debugging Python. The interface is a bit obtuse but should be familiar to anyone who has used similar tools, such as gdb. |
|||
|
|
|
The suggestion to install a signal handler is a good one, and I use it a lot. For example, bzr by default installs a SIGQUIT handler that invokes However, sometimes I need to debug a process that I didn't have the foresight to install the signal handler in. On linux, you can attach gdb to the process and get a python stack trace with some gdb macros. Put http://svn.python.org/projects/python/trunk/Misc/gdbinit in
It's not totally reliable unfortunately, but it works most of the time. Finally, attaching |
|||||||||||||||||||||
|
|
pyringe is a debugger that can interact with running python processes, print stack traces, variables, etc. without any a priori setup. While I've often used the signal handler solution in the past, it can still often be difficult to reproduce the issue in certain environments. |
|||||
|
|
You can try the faulthandler module. Install it using
at the beginning of your program. Then send SIGUSR1 to your process (ex: The module is now part of Python 3.3. For Python 2, see http://faulthandler.readthedocs.org/ |
||||
|
|
|
Take a look at the |
|||||
|
|
I am almost always dealing with multiple threads and main thread is generally not doing much, so what is most interesting is to dump all the stacks (which is more like the Java's dump). Here is an implementation based on this blog:
|
||||
|
|
|
If you're on a Linux system, use the awesomeness of Run your program with:
This instructs Now when you program hangs, switch into
See example session and more info here and here. |
||||
|
|
|
You can use PuDB, a Python debugger with a curses interface to do this. Just add
to your code and use Ctrl-C when you want to break. You can continue with |
|||||
|
|
I hacked together some tool which attaches into a running Python process and injects some code to get a Python shell. See here: https://github.com/albertz/pydbattach |
|||||
|
|
use the inspect module.
stack(context=1) Return a list of records for the stack above the caller's frame. I find it very helpful indeed. |
||||
|
|
|
On Solaris, you can use pstack(1) No changes to the python code are necessary. eg.
|
|||
|
I would add this as a comment to haridsv's response, but I lack the reputation to do so: Some of us are still stuck on a version of Python older than 2.6 (required for Thread.ident), so I got the code working in Python 2.5 (though without the thread name being displayed) as such:
|
||||
|
|
|
What really helped me here is spiv's tip (which I would vote up and comment on if I had the reputation points) for getting a stack trace out of an unprepared Python process. Except it didn't work until I modified the gdbinit script. So:
|
|||||||||||||||||
|
|
The traceback module has some nice functions, among them: print_stack:
|
|||||||||
|
|
It's worth looking at Pydb, "an expanded version of the Python debugger loosely based on the gdb command set". It includes signal managers which can take care of starting the debugger when a specified signal is sent. A 2006 Summer of Code project looked at adding remote-debugging features to pydb in a module called mpdb. |
|||||
|
You can also nicely format the stack trace, see the docs. Edit: To simulate Java's behavior, as suggested by @Douglas Leeder, add this:
to the startup code in your application. Then you can print the stack by sending |
|||||||||||||||||||||
|
|
There is no way to hook into a running python process and get reasonable results. What I do if processes lock up is hooking strace in and trying to figure out what exactly is happening. Unfortunately often strace is the observer that "fixes" race conditions so that the output is useless there too. |
|||||||||||||||||
|
|
I don't know of anything similar to java's response to SIGQUIT, so you might have to build it in to your application. Maybe you could make a server in another thread that can get a stacktrace on response to a message of some kind? |
|||
|
|
|
python -dv yourscript.py That will make the interpreter to run in debug mode and to give you a trace of what the interpreter is doing. If you want to interactively debug the code you should run it like this: python -m pdb yourscript.py That tells the python interpreter to run your script with the module "pdb" which is the python debugger, if you run it like that the interpreter will be executed in interactive mode, much like GDB |
|||||
|