44

I have a process that is called by another process which is called by another process and so on ad nauseam. It's a child process in a long tool chain.

This process is crashing.

I would like to catch this process in GDB to understand why it's crashing. However, the only way I can think of is:

  1. Start the original parent process in the commandline.
  2. Poll ps -C <name process I want to catch> and get the PID.
  3. Launch GDB, attached to that process's PID.

This is cumbersome but usually does the job. The problem is that the current failure runs very quickly, and by the time I capture the PID and launch GDB, it's already passed the failure point.

I would like to launch GDB and instead of:

(gdb) attach <pid>

I would like to do:

(gdb) attach <process name when it launches>

Is there any way to do this?


I am using GDB 7.1 on Linux.

7 Answers 7

31

Here is my script called gdbwait:

#!/bin/sh
progstr=$1
progpid=`pgrep -o $progstr`
while [ "$progpid" = "" ]; do
  progpid=`pgrep -o $progstr`
done
gdb -ex continue -p $progpid

Usage:

gdbwait my_program

Sure it can be written nicer but Bourne shell script syntax is painful for me so if it works then I leave it alone. :)

If the new process launches and dies too quickly, add 1 second delay in your own program for debugging.

3
11

On Mac OS X, you can use:

(gdb) attach --waitfor <process-name>

but this also is sometimes unable to capture processes that exit very quickly. I'm unsure if this is supported on any other platforms.

GDB Release Notes for Mac OS X v10.5 WWDC Seed

2
  • That's what I'm looking for. Unfortunately, it seems to be unsupported in my version (7.2) Jan 25, 2011 at 7:18
  • 2
    Use gdb-apple: sudo port install gdb-apple Jul 12, 2015 at 10:44
10

You can attach to a parent process and set follow-fork-mode child. This will make gdb debug child process instead of parent after forking. Also catch fork will be useful. This will make gdb stop after each fork. See docs.

2
  • But this can get pretty tedious if the process tree is deep (as it is in my case) Dec 8, 2010 at 10:44
  • @NathanFellman - One solution to the tediousness might be to write some Python extension code for gdb that automates this process. It could wake up after every caught fork, determine if that fork is relevant, then continue if it isn't. Oct 19, 2017 at 19:50
1

Not exactly what you expect, but it might help you in debugging.

valgrind --trace-children=yes your_program

will check and print memory errors in all children of the process, with stack trace and some detail about the error (eg. in case of double-free, you'd get the stack trace of the first free).

Also, you might make the crashing process generate a core dump, and debug this post-mortem. See this answer for details.

1

I've been facing a similar problem with something I'm trying to debug, and I came up with a solution using ldpreload, but after seeing Joeys answer I think I'll try that first. In case it's helpful to anyone else though here's the idea:

Create an LD_PRELOAD library to hook the exec* calls (there's plenty of guides on how to do this around, but if I do this I'll update my answer with the code), check the path used when passing through the exec* call, if it's our target then output a message with the PID to stderr and go into an infinite loop (with sleep to avoid massive CPU usage). Then you can attach with gdb and modify the register used in the loop to continue execution.

This may involve some inline ASM to make sure the compiler doesn't optimise the infinite loop in such a way that makes it hard to break out of. A more eloquent way of doing it would be to find a way to detect that gdb has attached then trigger a breakpoint ("asm("int3");" should do the trick on the latter).

1

One of the programs in Tom Tromey's gdb-helpers repository on GitHub is preattach, which will make gdb attach to the next process created with a given name. It requires the Linux systemtap program.

1

If the process crashes or exits before you can attach to it, you can patch its entrypoint so it loops at startup. And then unpause it from gdb or from the shell. This script works for x86_64 and i386 binaries:

#!/bin/bash
#
# Patch an ELF binary's entry point to pause it at startup.
# And patch it when loaded to unpause it.

set -oe pipefail
type readelf objdump dd awk grep > /dev/null

syntax()
{
    echo "syntax: `basename $0` --test  elf-file"
    echo "        `basename $0` --patch elf-file"
    echo "        `basename $0` --unpause PID"
    echo "        `basename $0` --addr    PID"
    exit 1
}

if [ "$1" = "--unpause" -o "$1" = "-u" -o "$1" = "--addr" ]; then
    PID="$2"; [ -n "$PID" ] || syntax
    BASE=$(head -1 /proc/$PID/maps | cut -d- -f1)
    START=$(readelf -h "/proc/$2/exe" | grep Entry | \
        awk 'match($0, /: *0x(.*)/, m) { print m[1] }' )
    if [ "$1" = "--addr" ]; then
        echo -e "(gdb)\nset {char}(0x$BASE+0x$START+1) = 0x02"
    else
        echo -en '\x02' | dd of=/proc/$PID/mem bs=1 \
            seek=$((0x$BASE+0x$START+1)) conv=notrunc status=none
    fi
    exit
elif [ "$1" = "--test" ]; then
    TEST=1
elif [ "$1" != "--patch" ]; then
    syntax
fi

shift; ELF="$1"; [ -n "$ELF" ] || syntax

START=$(readelf -h "$ELF" | grep Entry | \
    awk 'match($0, /: *0x(.*)/, m) { print m[1] }' )

objdump -d "$ELF" | grep -m1 " ${START}:" || true
[ "$TEST" != "1" ] || { echo -e "\nNow patch the binary using --patch.\n"; exit 1; }

echo -en "\xEB\xFE" | dd of="$ELF" bs=1 \
    seek=$((0x$START)) conv=notrunc status=none

objdump -d "$ELF" | grep -m1 " ${START}:" || true

echo
echo "Program will pause at startup. Unpause it as root with:"
echo "    $0 --unpause \`pidof `basename "$ELF"`\`"
echo "Or find the address to restore at runtime from gdb:"
echo "    $0 --addr \`pidof `basename "$ELF"`\`"
echo
1
  • 1
    This code is so hacky so it deserves one upvote. For the reader who is not fluent with intel opcodes, \xEB is the branch instruction. So \xEB\xFE generate an infinite loop (that consume a lot of CPU). Replacing \xFE by \x02 allows to jump to next instruction (and unblock the process). Sep 26, 2022 at 15:32

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.