8

I have been working for the last few weeks trying to track down a really difficult bug that crashes my application. First, the application was crashing on the assign of a std::string, then during the free of a local variable.

After careful inspection of the code, there was no reason for it to crash at these locations; however, it always crashed while trying to free an invalid pointer (i.e. a pointer that pointed to invalid memory). And I have no idea why this pointer was not pointing to the right location.

I suspect that the issue has to do with a memory corruption problem or pointer corruption problem of some sort. The problem is that I can't visually track it down....yet. I have no idea where to start looking in the code, and there are thousands of lines of code to go through so this does not seem like a realistic approach to the problem.

So in comes Valgrind...

A tool that I have depended upon many a time to find issues within the code that may lead to a crash of this type. However, this time it has come up empty handed! I do not see any errors in valgrind when the problem occurs and so hence the reason for me asking this question.

Are there any other applications that can complement valgrind and help find issues in the code that may cause a crash mentioned above?

Thanks!

2
  • You could try posting a minimal, compilable code sample that reproduces the problem. I'm willing to bet the act of trying to write such a code sample will make the problem obvious to you. If not, we are more than happy to help. Feb 18, 2010 at 14:48
  • Would you like to see where the code is crashing (i.e. a sample of where it crashes)? because there's no reason for it to crash there as I mentioned, but I don't mind posting it
    – bbazso
    Feb 18, 2010 at 15:01

8 Answers 8

6

I assume you're using valgrind's memcheck tool, which is what it is famous for. Since you are using valgrind already you might also try running your program through valgrind --tool=exp-sgcheck (formerly exp-ptrcheck), which is an experimental tool that is designed to catch certain types of errors that memcheck will miss, including access checks for stack and global arrays, and use of pointers that happen to point to a valid object but not the object that was intended. It does this by using a completely different mechanism, essentially tracking each pointer into memory rather than tracking the memory itself, and through use of heuristics.

Be aware that the tool is experimental, but you may find that it catches something significant. Currently it does not yet support OS X or non-Intel processors.

3
  • I tried it out, but I keep getting a: sysno == 233 exp-ptrcheck: the 'impossible' happened: unhandled syscall Any ideas?
    – bbazso
    Feb 18, 2010 at 18:33
  • Tried it out and then it moved down one to 232. :) Where did you find the mapping between syscall no and what it is?
    – bbazso
    Feb 18, 2010 at 21:31
  • /usr/include/asm-i386/unistd.h (or asm-x86_64 for x86_64). I guess you can see why it is still experimental...
    – mark4o
    Feb 18, 2010 at 21:48
5

In my experience, coverity and purify have founds such kind of errors than valgrind didn't (in fact all found problems which weren't seen by the others).

But sometimes no tool give an hint and you have to dig more, add instrumentation, play with breakpoints on "modify memory at address", try to simply the testcase which fails and so on to find out the root cause. That's can be very painful.

3
  • I agree with the debugger comment. Sometimes it's just best to set a break point on the memory that gets blown away and wait.
    – deft_code
    Feb 18, 2010 at 15:36
  • With gdb, how can I set a breakpoint on memory to see when it's written to?
    – bbazso
    Feb 18, 2010 at 15:59
  • the watch command, potentially along with a condition on what you consider invalid Feb 18, 2010 at 16:05
3

My experience is that often this sort of problem is caused by a heap overflow. Electric Fence is a relatively simple allocation debugging tool I like to use. Its main use is as a dynamic analysis tool to check for heap overflows, a complement to "-fstack-protector-all" which checks for stack overflows.

More links to efence stuff.

2

Is it possible some stack corruption is occurring? If so, try enabling stack canaries with the -fstack-protector-all option, assuming you are using g++.

Other than that, have you cranked up warning flags to help identify suspicious code?

1
  • I tried the flag -fstack-protector-all and I also tried libsafe and both came up empty handed ? :(
    – bbazso
    Feb 18, 2010 at 18:40
2

In my opinion, using a debugger with "reverse debugging" capabilities could help. You would be able to step back in time and hopefully find out what was the real source of the problem.

Here are a couple of links:

http://www.gnu.org/software/gdb/news/reversible.html

http://undo-software.com/ (which apparently is free for non-commercial applications)

1

You didn't specify the platform, but I can recommend Gimpel PC-lint as an excellent static analysis tool (don't be fooled by the name!). They also offer FlexeLint for other platforms, but I have no personal experience of that product.

1
  • Gimpel Software is a great outfit. +1 Feb 19, 2010 at 2:43
0

Have you tried using lint, flexlint or cppcheck. These may help identify a problem.

If you know what area of memory is being corrupted have you tried marking this memory as protected. This may mask your problem and not help at all but if it still crashes the point at where the memory is modified will help resolve your problem.

0
0

If valgrind can identify the bad pointer being passed to free(), you could try running the program under DDD, which can set a hardware watchpoing on the memory location and halt the program when it is getting a bad value. If the pointer is getting changed a lot you may have to write some code around malloc and free to keep track of which values are good and bad.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.