1

I am trying to debug a multi-process solution on FreeBSD. When the system/appliance experienced a hang like scenario, we forced a kernel dump through 'sysctl debug.panic=1'. The intention was to capture the state of all processes at the same point in time. However, I am not able to look into the thread stacks of userspace applications. Using 'ps', I am able to list all userspace processes/threads but not able to set their stack frame and unwind using 'bt'.

Is it possible to achieve something like what I am attempting to perform? I have seen OpenVMS debugger (IIRC even windbg) allowing one to peek into userspace threads.

2 Answers 2

1

Use DDB. It supports tracing of threads. See this article. The same article also names kgdb commands to trace userspace threads. But those are not found in the manual page. :-(

0
1

In DDB "bt/u" will trace userland portion of thread's stack. See "man 4 ddb". That, combined with textdump may be enough.

If all you have to work with is core, things get a bit more complicated.

In kgdb "info threads" will list all threads that were running at the time of kernel crash. After that "thread X" followed by "bt" will give you in-kernel portion of thread's stack.

Getting userland portion of the application will be harder. Easiest way to do that would probably be to modify gcore application so that it uses libkvm to dig into VM structures associated with a given process and essentially reconstruct process' coredump. It is possible, but I don't think there's a ready-to-use solution at the moment.

2
  • They have removed ddb on the appliance, can it get harder... Let me first find a working ddb and see how far I can go. I am using cross-compiled gdb on linux
    – deepsnore
    Apr 2, 2012 at 14:39
  • Regular GDB may not be all that useful for debugging FreeBSD kernel cores. First, not all kernel core formats are 'standard'. FreeBSD's KGDB is linked with libkvm which knows how to deal with minidumps (which is the default these days), for instance. At the very least you will need to disable minidumps with 'sysctl debug.minidump=0' so that produced cores are in somewhat normal ELF format. Even then, the core will have physical memory dumped and GDB will need libkvm in order to translate kernel virtual addresses into physical ones.
    – ArtemB
    Apr 2, 2012 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.