I am seeing high cpu utilization while running java program. I found jstack can help to debug the things where it is stuck and eating the CPU. I got the output with jstack and found there are few threads are in BLOCKED state.
Here I have used java 1.7 and to confirm the load issue I have replaced the java with older 1.6 on the same OS ie CentOS 6.3 but the issue still remain, since have tested with combination of java 1.6 (1.7 as well) and centos 5.4 which works great and using less cpu.
There is nothing has been changed except the OS which is centos 6.3 only getting load issue.
How can I find the relevant code which pointed threads as BLOCKED into jstack output, following are the output from my findings
OS - CentOS 6.3 x86_64 (2.6.32-279.el6.x86_64)
CPU core - 8
Ram - 16GB
Java version - "1.7.0_17"
jstack output:
Deadlock Detection:
No deadlocks found.
none
null_check
null_assert
range_check
class_check
array_check
intrinsic
bimorphic
unloaded
uninitialized
unreached
unhandled
constraint
div0_check
age
predicate
loop_limit_check
Thread 4514: (state = BLOCKED)
Locked ownable synchronizers:
- None
Thread 4493: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Interpreted frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=177 (Interpreted frame)
Locked ownable synchronizers:
- None
Thread 4491: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Interpreted frame)
Locked ownable synchronizers:
- None
Java Info Output:
Java System Properties:
java.runtime.name = Java(TM) SE Runtime Environment
java.vm.version = 23.7-b01
sun.boot.library.path = /usr/java/jdk1.7.0_17/jre/lib/amd64
hadoop.root.logger = INFO,console
java.vendor.url = http://java.oracle.com/
java.vm.vendor = Oracle Corporation
path.separator = :
file.encoding.pkg = sun.io
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
sun.os.patch.level = unknown
sun.java.launcher = SUN_STANDARD
user.country = US
user.dir = /home/ravi
java.vm.specification.name = Java Virtual Machine Specification
java.runtime.version = 1.7.0_17-b02
java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
os.arch = amd64
java.endorsed.dirs = /usr/java/jdk1.7.0_17/jre/lib/endorsed
line.separator =
VM Flags:
-Xmx4000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=//opt/cloudera/parcels/CDH-4.2.1-1.cdh4.2.1.p0.5/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=//opt/cloudera/parcels/CDH-4.2.1-1.cdh4.2.1.p0.5/lib/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=//opt/cloudera/parcels/CDH-4.2.1-1.cdh4.2.1.p0.5/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xms256m -Dhadoop.security.logger=INFO,NullAppender