3

I was trying to figure out how to know if the threads of a process have been deadlocked on Unix/Linux machine? Also, is there a command for knowing what stage (or status) a process is in? If you know of any tools, please suggest. Thank you.

3 Answers 3

2

Thanks to /proc/<pid>/syscall, this is how I ended up implementing a quick and dirty processes futex(op=FUTEX_WAIT) scanner.

#!/bin/bash
#
# Find all processes that are executing a futex(2) call with op=FUTEX_WAIT
# In some cases this can be helpful in finding deadlock-ed processes.
#

test ! $UID -eq 0 && echo -e "WARNING: Not running as root, only processes for this user are being scanned\n" >&2;
pids=$(ps -u $UID -opid --no-headers)

for pid in $pids; do
        cat /proc/$pid/syscall |

        awk "{if (\$1 == 202 && \$3 == \"0x0\") {
                print $pid
        }}";

        # $1 is the syscall, we compare to 202 which is the futex call
        # See: /usr/include/asm/unistd.h

        # $2 is the 1st param, $3 is the 2nd param, etc
        # We compare the second param to 0x0 which is FUTEX_WAIT
        # See: /usr/include/linux/futex.h
done
1

Try to use a tool that traces system calls, e.g strace on Linux or tusc on HP-UX. When a deadlock occured, you should see the process hang in the blocking call. That's not a positive proof, though. It might be a regular block. You then need to determine whether the block may be resolved some time or not. This requires knowledge of the resource the process is waiting for.

Example

On RHEL4 there exists a... peculiarity... that may cause ctime to deadlock. Find an example program exhibiting this behaviour below:

#include <sys/time.h>
#include <time.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

volatile char *r;

void handler(int sig)
{
    time_t t;

    time(&t);
    r = ctime(&t);
}

int main()
{
    struct itimerval it;
    struct sigaction sa;
    time_t t;
    int counter = 0;

    memset(&sa, 0, sizeof(sa));
    sa.sa_handler = handler;
    sigaction(SIGALRM, &sa, NULL);

    it.it_value.tv_sec = 0;
    it.it_value.tv_usec = 1000;
    it.it_interval.tv_sec = 0;
    it.it_interval.tv_usec = 1000;
    setitimer(ITIMER_REAL, &it, NULL);

    while(1) {
        counter++;
        time(&t);
        r = ctime(&t);
        printf("Loop %d\n",counter);
    }

    return 0;
}

This will typically enter a deadlock after a couple thousand iterations. Now, attach strace like so

strace -s4096 -p<PID>

where PID is the program's process id. You'll see the program hang in a call with FUTEX_WAIT in the parameters. (I cannot quote the whole line as I currently don't have access to a RHEL4 machine, please excuse that).

2
  • It would be very helpful if this could be scripted somehow. I read the man page of strace but couldn't find a way to print only the current/latest syscall. If that was possible, I could loop on all processes and check for a futex call with FUTEX_WAIT. Nov 30, 2010 at 17:20
  • @Amr Mostafa. strace -s4096 -p$PID 2>&1 | grep -v -E -e "^Process" -e "^restart_syscall" | head -c12 may get you started. Both the constant 12 and the regular expressions may need some love, but the result seems promising.
    – dennycrane
    Dec 1, 2010 at 1:24
-1

UNIX guarantees that OS processes never gets in deadlock. No such guarantees are, however, made for user defined processes. As far as I know, there is no direct way of determining whether a process has been deadlocked.

That said, you could determine the process state by ps -o pid,uname,command,state,stime,time. man ps shows more detailed explanation on process state codes too.

1
  • There's nothing in UN*X that guarantees OS/kernel processes/threads do not deadlock. If they do it's a bug, admittedly, but the OS/kernel design / API don't make such deadlocks impossible.
    – FrankH.
    Apr 11, 2012 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.