I have a application which uses Spring DefaultMessageListenerContainer and IBM mq as the message broker, after running application for certain time, the cpu usage spiked with other threads with similar cpu usage. Below is one of the messageListenerContainer thread from thread dump with top command.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29410 tomcat 20 0 10.4g 4.0g 20320 R 25.0 12.6 610:36.22 msgListenerCont
"msgListenerContainer" #265306 prio=5 os_prio=0 cpu=36635917.55ms elapsed=3263601.16s tid=0x00007f8804046800 nid=0x72e2 runnable [0x00007f87abe79000]
java.lang.Thread.State: RUNNABLE
at java.lang.ThreadLocal$ThreadLocalMap.set(java.base@11.0.7/ThreadLocal.java:487)
at java.lang.ThreadLocal.set(java.base@11.0.7/ThreadLocal.java:222)
at com.ibm.msg.client.commonservices.locking.TraceableLock.incrementDepth(TraceableLock.java:99)
at com.ibm.msg.client.commonservices.locking.TraceableLock.lock(TraceableLock.java:90)
at com.ibm.msg.client.commonservices.locking.TraceableReentrantLock.lock(TraceableReentrantLock.java:73)
at com.ibm.mq.jmqi.remote.util.HconnLock.lock(HconnLock.java:83)
at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.requestMutex(RemoteProxyQueue.java:367)
at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.requestMessagesReconnectable(RemoteProxyQueue.java:1046)
- locked <0x00000007166f9d48> (a com.ibm.mq.jmqi.remote.impl.RemoteSession$RequestMessagesMutex)
at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.requestMessages(RemoteProxyQueue.java:717)
at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.flushQueue(RemoteProxyQueue.java:1911)
at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.proxyMQGET(RemoteProxyQueue.java:2783)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiGetInternalWithRecon(RemoteFAP.java:6103)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiGetInternal(RemoteFAP.java:5992)
at com.ibm.mq.jmqi.internal.JmqiTools.getMessage(JmqiTools.java:1371)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiGet(RemoteFAP.java:5957)
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiGet(InterceptedJmqiImpl.java:1341)
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiGet(ESEJMQI.java:602)
at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.getMsg(WMQConsumerShadow.java:1795)
at com.ibm.msg.client.wmq.internal.WMQSyncConsumerShadow.receiveInternal(WMQSyncConsumerShadow.java:228)
at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.receive(WMQConsumerShadow.java:1461)
at com.ibm.msg.client.wmq.internal.WMQMessageConsumer.receive(WMQMessageConsumer.java:674)
at com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receiveInboundMessage(JmsMessageConsumerImpl.java:1051)
at com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receive(JmsMessageConsumerImpl.java:667)
at com.ibm.mq.jms.MQMessageConsumer.receive(MQMessageConsumer.java:209)
at org.springframework.jms.support.destination.JmsDestinationAccessor.receiveFromConsumer(JmsDestinationAccessor.java:132)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:418)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:303)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1189)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1179)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1076)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
From above thread stack trace it looks to me that thread is stuck doing something with ThreadLocalMap.set method, but other than that I can't find why it is using so much cpu.
spring.jms.listener.receiveTimeout
to a higher value (default is 1 second)? If you use the MQ JMS Spring Boot Start this would be done automatically, see JMS Polling Listener Timer configuration.