52

I am running Ubuntu server edition and I wanted to take a thread dump of Tomcat.

So, I first tried to find out which PID tomcat uses:

$ jps -l
5809 sun.tools.jps.Jps

But it's not there?

So, I used top instead and found out the PID 5730.

Then I called jstack to get the thread dump:

$ sudo jstack -l 5730
5730: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

What's going on? :-(

I already tried to export CATALINA_TMPDIR as described in Jstack and Jstat stopped working with upgrade to JDK6u23 but that didn't change anything:

$ export CATALINA_TMPDIR=/tmp
$ sudo /etc/init.d/tomcat6 restart
 * Stopping Tomcat servlet engine tomcat6
   ...done.
 * Starting Tomcat servlet engine tomcat6
   ...done.
$ sudo jstack -l 5934 // new PID after restart
5934: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

Update:

I also tried sudo -u tomcat6 jstack -l -F 5730 > threaddumpexceptions2.txt but it only gives me tons of exceptions on the console.

13

10 Answers 10

78

I got it working by doing two things:

  1. Changed call to: sudo -u tomcat6 jstack -J-d64 -m pid
  2. Replaced OpenJDK with Sun's original sun-6-jdk and sun-6-jre packages

Explanation for part 1: I switched to 64-bit mode, used sudo and run the command as Tomcat user.

Note: Part 2 might not be necessary. For some users it seems like part 1 is enough. In fact, try to add just the sudo command first. It might already do the trick.

5
  • 34
    Placing sudo -u <javaProcUsername> in front of the command did the trick for me.
    – Jono
    Commented Jun 25, 2012 at 15:35
  • 4
    By the way - just by running as the same user, I find I eliminate this error - in my case attempting to connect with yourkit, but I believe the same principle applies. Being in Sun JDK vs. OpenJDK, or specifying 64 vs. 32 bit seems unnecessary for me to fix it.
    – David W
    Commented Mar 14, 2014 at 20:22
  • 1
    Running the jstack command with the same user under which the jvm was running fixed the issue for me. Thanks @DavidW ! Commented Jul 30, 2014 at 9:49
  • The -J-d64 from root did the trick for me, I tried every combination, correct user, sudo, root. With and without native frames (-m) but it started working when I gave the -J-d64 flag, I also needed -F to get around the process being unresponsive.
    – feldoh
    Commented Oct 6, 2015 at 10:15
  • After switching to tomcat's user, the -F flag was actually making it fail for me.
    – Alkanshel
    Commented Mar 22, 2017 at 21:58
34

I think you need to run jstack as the same user that runs the Tomcat process. Note also that jps only returns processes for the current user. You would get the pid for the Tomcat process by running jps with sudo or as the Tomcat process user.

This bug report may also be useful: https://bugs.launchpad.net/ubuntu/+source/sun-java6/+bug/597098

5
  • I can't switch to tomcat6 user because it's not actually a "real" unix account on the system. So su tomcat6 doesn't work. I just tried sudo jps -l but it still only gives me 6493 sun.tools.jps.Jps
    – Timo Ernst
    Commented Sep 14, 2011 at 20:42
  • I also tried sudo -u tomcat6 jstack -l -F 5730 > threaddumpexceptions2.txt but it only gives me tons of exceptions on the console: dl.dropbox.com/u/17844821/zeug/threaddump2.txt - The threaddump itself looks like this: dl.dropbox.com/u/17844821/zeug/threaddumpexceptions2.txt - Doesn't look right to me.
    – Timo Ernst
    Commented Sep 14, 2011 at 20:57
  • @valmar: try this to become tomcat6: switches to root: sudo su - from root you you can switch to any user su tomcat6 Commented Sep 21, 2011 at 21:47
  • @Cd-MaN No effect. If I do that, it gives me this (the slashes represents newlines since I cannot post newline characters here in comments): $ whoami // valmar // $ sudo su - // # whoami // root // # su tomcat6 // # whoami // root
    – Timo Ernst
    Commented Sep 22, 2011 at 13:39
  • 2
    @valmar: this probably means that you do't have a shell set for the given user (tomcat6) - or most precisely you have the shell set to something like /bin/false. To solve this, use su -p tomcat6 from under the root account (this preserves the current shell). Commented Oct 20, 2011 at 7:53
4

@Valmar, I find the same topic post here. Unable to get thread dump? Any ideas why my app blocks?

It seems the workaround is sudo -u tomcat6 kill -3 <pid>.

5
  • Nop, that didn't do the trick. When I do that, nothing happens.
    – Timo Ernst
    Commented Sep 24, 2011 at 19:50
  • 1
    if you are using 64bit, you need to use jstack -J-d64 -m pid, also try to add sudu -u tomcat6 . download.oracle.com/javase/6/docs/technotes/tools/share/…
    – Clark Bao
    Commented Sep 25, 2011 at 11:48
  • Hm, that only gives me a load of exceptions on the console: dl.dropbox.com/u/17844821/zeug/console64.txt
    – Timo Ernst
    Commented Sep 25, 2011 at 12:19
  • 1
    If I were you,I will try to use another jdk not openjdk. Maybe it's due to openjdk.
    – Clark Bao
    Commented Sep 26, 2011 at 11:23
  • Interesting. I replaced openjdk-6-jre with sun-java6-jre and now jstack seems to work properly. jps still doesn't show tomcat and its PID though. @Clark Thanks for your help so far! If you post your suggestion to replace the JRE as a new, separate answer, I'll mark it as "the right answer".
    – Timo Ernst
    Commented Sep 27, 2011 at 15:07
3

Try to switch to process user and then use jstack:

sudo -u {process user} jstack > dump

1

This also worked for me:

sudo -u tomcat6 kill -3 pid

It looks like nothing happens but when you look in the logs the stacks are there. They look like exceptions if your not expecting them.

1

I find it useful to use something like 'ps -eo pid,user,command | grep java' to find the actual java command being used, then use the directory to find the matching jstack etc.

# ps -eo user,command | grep '[j]ava' | cut -d' ' -f1
someuser /usr/lib/jvm/java/bin/java

# /usr/lib/jvm/java/bin/java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

So its 64-bit, running as 'someuser'. su to that user and run run jstack etc. from that same directory. (ie. /usr/lib/jvm/java/bin/jstack

Useful when you're on a server with various different installations / implementations of Java.

1

For Tomcat users having this issue, check your catalina.out log file.

I was having the same problems "22693: Unable to open socket file: target process not responding or HotSpot VM not loaded". I gave up and was trying to find anything about what happened before it locked up, but then there was the jstack output in the log file.

0

I had same problem, but none of below solution worked for me:

jstack <pid>
jstack -J-d64 -m <pid>
sudo -u <user> jstack ...

I finally upgraded JDK from jdk1.6.0_24 to jdk1.7.0_67 and every things worked.

0

For those finding this answer six years after it was asked and they are on CentOS 7 note that SELinux will stop jmap from writing the heap dumps as, out of the box, it will deny writing to a socket unless the directory has the type tomcat_tmp_t.

In my case, I write my heap dumps under /usr/share/tomcat/.jmap. This directory is owned by my runtime user.

Therefore to allow jmap to write to this directory:

semanage fcontext -a -t tomcat_tmp_t "/usr/share/tomcat/.jmap(/.*)?"
restorecon /usr/share/tomcat -vR

Which allows us to then run, as our tomcat user:

jmap -dump:format=b,file=/usr/share/tomcat/.jmap/tomcat-`date +%s`.bin <pid>
0

My problem was, I ran the process by the terminal. Then for testing Its jstack, I paused the process by ctrl+z. The process was not able to respond. To solve this problem, I resumed the process by fg and checked its jstack by another terminal.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.