文章目录
  1. 1. 问题
  2. 2. 排查问题
  3. 3. 解决方案
  4. 4. 参考

问题

最近部署的应用监控提示一台机器CPU的load特别高,而部署相同应用的其他机器却没有任何问题。本文就记录下来解决该问题的整个过程。

排查问题

首先是上线之后top命令,发现CPU飙高的正是部署在机器上的Java进程。

  1. jstack打印Java堆栈信息,保留现场

    jstack pid > jstack.log
    
  2. 查看堆栈信息,同事提醒说,一般CPU飙高主要查看堆栈Runnale的线程。但是漫无目标去查看,很难定位到问题。
  3. 通过 top -H -p PID命令输出占用cpu过高的线程
  4. 将占用CPU过高的线程ID转换为16进制

    printf %x pid
    
  5. 去原来保存的堆栈文件中查找对应线程,就是出问题的线程。最终我们发现问题线程如下:
"HSFBizProcessor-8-thread-2300" daemon prio=10 tid=0x00007fbb4813e000 nid=0x5c94 runnable [0x00007fbb3a8b0000]
   java.lang.Thread.State: RUNNABLE
    at java.util.HashMap.getEntry(HashMap.java:469)
    at java.util.HashMap.containsKey(HashMap.java:453)
    at java.util.HashSet.contains(HashSet.java:201)
    at com.ttpod.alimusic.BaseClient.saveLoseDataLog(BaseClient.java:201)
    at com.ttpod.alimusic.client.SongClient.getAudioForSongs(SongClient.java:103)
    at com.ttpod.alimusic.client.SongClient$$FastClassBySpringCGLIB$$4c35f712.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
    at com.ttpod.alimusic.aspect.LogAspect.doLog(LogAspect.java:49)
    at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
    at com.ttpod.alimusic.client.SongClient$$EnhancerBySpringCGLIB$$931c77b8.getAudioForSongs(<generated>)
    at com.ttpod.alimusic.dao.SongFileALiDao.getSongFiles(SongFileALiDao.java:31)
    at com.ttpod.cache.SongFileListCache.lazyLoad(SongFileListCache.java:38)
    at com.ttpod.cache.LocalListCache.getList(LocalListCache.java:57)
    at com.ttpod.song.service.SongFileServiceImpl.getSongFilesByIds(SongFileServiceImpl.java:86)
    at com.ttpod.song.service.SongServiceImpl.fillUrl(SongServiceImpl.java:424)
    at com.ttpod.song.service.SongServiceImpl.loadSongFiles(SongServiceImpl.java:117)
    at sun.reflect.GeneratedMethodAccessor439.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.taobao.hsf.remoting.provider.ProviderProcessor.handleRequest0(ProviderProcessor.java:471)
    at com.taobao.hsf.remoting.provider.ProviderProcessor.handleRequest(ProviderProcessor.java:152)
    at com.taobao.hsf.remoting.server.RPCServerHandler.handleRequest(RPCServerHandler.java:43)
    at com.taobao.hsf.remoting.server.RPCServerHandler.handleRequest(RPCServerHandler.java:21)
    at com.taobao.hsf.remoting.netty.server.NettyServerHandler$HandlerRunnable.run(NettyServerHandler.java:109)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

后面看到一个更好的排查方案:show-busy-java-threads.sh

解决方案

将HashMap换成线程安全的ConcurrentHashMap.

至于关于HashMap线程不安全问题,可以查看左耳朵耗子的分析 疫苗:Java HashMap的死循环

参考

  1. jstack命令(Java Stack Trace)
文章目录
  1. 1. 问题
  2. 2. 排查问题
  3. 3. 解决方案
  4. 4. 参考