132

I have the following stack trace. Is it possible to make out anything useful from this for debugging?

Program received signal SIGSEGV, Segmentation fault.
0x00000002 in ?? ()
(gdb) bt
#0  0x00000002 in ?? ()
#1  0x00000001 in ?? ()
#2  0xbffff284 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) 

Where to start looking at the code when we get a Segmentation fault, and the stack trace is not so useful?

NOTE: If I post the code, then the SO experts will give me the answer. I want to take the guidance from SO and find the answer myself, so I'm not posting the code here. Apologies.

2
  • Probably your program jumped off into the weeds - can you recover anything from the stack pointer?
    – Carl Norum
    Mar 21, 2012 at 17:36
  • 3
    Another thing to consider is if the frame pointer is set correctly. Are you building without optimizations or passing a flag like -fno-omit-frame-pointer? Also, for memory corruption, valgrind might be a more appropriate tool, if it's an option for you.
    – FatalError
    Mar 21, 2012 at 17:36

6 Answers 6

166

Those bogus adresses (0x00000002 and the like) are actually PC values, not SP values. Now, when you get this kind of SEGV, with a bogus (very small) PC address, 99% of the time it's due to calling through a bogus function pointer. Note that virtual calls in C++ are implemented via function pointers, so any problem with a virtual call can manifest in the same way.

An indirect call instruction just pushes the PC after the call onto the stack and then sets the PC to the target value (bogus in this case), so if this is what happened, you can easily undo it by manually popping the PC off the stack. In 32-bit x86 code you just do:

(gdb) set $pc = *(void **)$esp
(gdb) set $esp = $esp + 4

With 64-bit x86 code you need

(gdb) set $pc = *(void **)$rsp
(gdb) set $rsp = $rsp + 8

Then, you should be able to do a bt and figure out where the code really is.

The other 1% of the time, the error will be due to overwriting the stack, usually by overflowing an array stored on the stack. In this case, you might be able to get more clarity on the situation by using a tool like valgrind

12
  • 6
    @George: gdb executable corefile will open gdb with the executable and core file, at which point you can do bt (or the above commands followed by bt)...
    – Chris Dodd
    Mar 27, 2014 at 18:58
  • 2
    @mk.. ARM doesn't use the stack for return addresses -- it uses the link register instead. So it generally doesn't have this problem, or if it does, it is usually due to some other stack corruption.
    – Chris Dodd
    Apr 17, 2015 at 22:10
  • 2
    Even in ARM, I think, all the General purpose registers and LR are stored in stack before the called function starts executing. Once the function finishes, the value of LR is poped into PC and hence the function returns. So If stack is corrupted, we can see a wrong value is PC right? In this case may be adjusting stack pointer will lead to appropriate stack and help to debug the issue. What do you think? pls let me know your thoughts. Thank you.
    – Sandeep
    Apr 20, 2015 at 2:37
  • 1
    What means bogus?
    – Danny Lo
    May 17, 2017 at 11:12
  • 6
    ARM is not x86 -- its stack pointer is called sp, not esp or rsp, and its call instruction stores the return address in the lr register, not on the stack. So for ARM, all you really need to undo the call is set $pc = $lr. If $lr is invalid, you have a much harder problem to unwind.
    – Chris Dodd
    Oct 2, 2018 at 4:38
50

If the situation is fairly simple, Chris Dodd's answer is the best one. It does look like it jumped through a NULL pointer.

However, it is possible the program shot itself in the foot, knee, neck, and eye before crashing—overwrote the stack, messed up the frame pointer, and other evils. If so, then unraveling the hash is not likely to show you potatoes and meat.

The more efficient solution will be to run the program under the debugger, and step over functions until the program crashes. Once a crashing function is identified, start again and step into that function and determine which function it calls causes the crash. Repeat until you find the single offending line of code. 75% of the time, the fix will then be obvious.

In the other 25% of situations, the so-called offending line of code is a red herring. It will be reacting to (invalid) conditions set up many lines before—maybe thousands of lines before. If that is the case, the best course chosen depends on many factors: mostly your understanding of the code and experience with it:

  • Perhaps setting a debugger watchpoint or inserting diagnostic printf's on critical variables will lead to the necessary A ha!
  • Maybe changing test conditions with different inputs will provide more insight than debugging.
  • Maybe a second pair of eyes will force you to check your assumptions or gather overlooked evidence.
  • Sometimes, all it takes is going to dinner and thinking about the gathered evidence.

Good luck!

3
  • 15
    If a second pair of eyes are not available then rubber ducks are well proven as alternatives.
    – Matt
    Mar 21, 2012 at 18:52
  • 2
    Writing off the end of a buffer can do it, too. It might not crash where you write off the end of the buffer, but when you step out of the function, then it dies.
    – phyatt
    Sep 23, 2016 at 20:15
  • May be useful: GDB: Automatic 'Next'ing
    – user202729
    Oct 12, 2018 at 4:22
33

Assuming that the stack pointer is valid...

It may be impossible to know exactly where the SEGV occurs from the backtrace -- I think the first two stack frames are completely overwritten. 0xbffff284 seems like a valid address, but the next two aren't. For a closer look at the stack, you can try the following:

gdb$ x/32ga $rsp

or a variant (replace the 32 with another number). That will print out some number of words (32) starting from the stack pointer of giant (g) size, formatted as addresses (a). Type 'help x' for more info on format.

Instrumenting your code with some sentinel 'printf''s may not be a bad idea, in this case.

4
  • Incredibly helpful, thank you -- I had a stack that only went back three frames and then hit "Backtrace stopped: previous frame identical to this frame (corrupt stack?)"; I've done something exactly like this in code in a CPU exception handler before, but couldn't remember other than info symbol how to do this in gdb.
    – leander
    Mar 8, 2013 at 19:05
  • 26
    FWIW on 32-bit ARM devices: x/256wa $sp =)
    – leander
    Mar 8, 2013 at 19:05
  • 2
    @leander Could you tell me what is X/256wa? I need it for 64-bit ARM. In general it will be helpful if you can explain what is it.
    – Sandeep
    Apr 17, 2015 at 12:15
  • 5
    Per the answer, 'x'=examine memory location; it prints out a number of 'w'=words (in this case, 256), and interprets them as 'a'=addresses. There's more info in the GDB manual at sourceware.org/gdb/current/onlinedocs/gdb/Memory.html#Memory .
    – leander
    Apr 17, 2015 at 23:18
7

Look at some of your other registers to see if one of them has the stack pointer cached in them. From there, you might be able to retrieve a stack. Also, if this is embedded, quite often stack is defined at a very particular address. Using that, you can also sometimes get a decent stack. This all assumes that when you jumped to hyperspace, your program didn't puke all over memory along the way...

4

If it's a stack overwrite, the values may well correspond to something recognisable from the program.

For example, I just found myself looking at the stack

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x000000000000342d in ?? ()
#2  0x0000000000000000 in ?? ()

and 0x342d is 13357, which turned out to be a node-id when I grepped the application logs for it. That immediately helped narrow down candidate sites where the stack overwrite might have occurred.

0

funny...we had the exact same thing going on with a driver in an old C app here. the top 2 stack trace value pointers in hex were data bytes being read in off the port. I just happened to notice one because it was familiar.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.