诡异的JVM堆外内存泄漏

叔牙 • 2023-01-02 • 云技术社区 • 269 阅读

一、现象

● 报警详情: MEM usage above 90% (current value: 0.9731329333728482)
● 堆外metaspace内存占用高达3GB多
● 机器内存耗尽，宕机

二、概念

元空间是jdk1.8开始取代永久代的内存模型,被jvm使用受操作系统管辖的直接内存区域。

jdk1.7内存结构：

jdk1.8内存结构:

三、排查思路

● metaspace增长是逐渐增多，增长速度不均匀，考虑是接口调用或mq
● metaspace主要存放类信息，所以主要怀疑:动态类生成类库的使用
● 项目中使用的第三方库：fastjson,dubbo,aspectj,spring
● 首先排除了fastjson,dubbo,spring，因为足够稳定和靠谱
● 但是预发环境和测试环境，依旧无法重现问题

四、排查过程

● jmap查看JVM对象信息,发现大量和反射相关对象被生成

admin@ip-xxx:~/logs/xxx$ jmap -histo pid

 num     #instances         #bytes  class name
----------------------------------------------
   1:        836613      102185264  [C
   2:        128213       30887816  [B
   3:        546707       13120968  java.lang.String
   4:        390652       12500864  java.util.HashMap$Node
   5:         58502       10908984  [I
   6:        121161       10662168  java.lang.reflect.Method
   7:        149994       10117912  [Ljava.lang.Object;
   8:         61139        6432184  [Ljava.util.HashMap$Node;
   9:        160882        5148224  java.util.concurrent.ConcurrentHashMap$Node
  10:        172682        3822600  [Ljava.lang.Class;
  11:         58938        2829024  java.util.HashMap
  12:         24263        2688136  java.lang.Class
  13:        110260        2646240  java.util.ArrayList
  14:         65896        2635840  java.util.LinkedHashMap$Entry
  15:         41654        2252544  [Ljava.lang.String;
  16:          2459        1894392  [Ljava.util.concurrent.ConcurrentHashMap$Node;
  17:         24752        1782144  java.lang.reflect.Field
  18:         73982        1775568  java.lang.StringBuilder
  19:         29437        1648472  java.util.LinkedHashMap
  20:         51507        1648224  java.lang.ref.WeakReference
  21:         34020        1632960  org.aspectj.weaver.reflect.ShadowMatchImpl
  22:         30742        1229680  com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl$1
  23:         76420        1222720  java.lang.Integer
  24:         34020        1088640  org.aspectj.weaver.patterns.ExposedState
  25:         12658        1012640  java.lang.reflect.Constructor
  26:         24377         975080  java.util.HashMap$KeyIterator
  27:         59090         945440  java.lang.Object
  28:         23290         931600  java.lang.ref.SoftReference
  29:         18518         919184  [Ljava.lang.reflect.Method;
  30:         35328         847872  com.dianping.cat.internal.shaded.io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry
  31:         24738         791616  com.sun.xml.internal.stream.events.CharacterEvent
  32:         13612         762272  sun.nio.cs.UTF_8$Encoder
  33:         31471         755304  io.termd.core.term.Feature
  34:         23095         739040  com.sun.org.apache.xerces.internal.xni.QName
  35:         10412         666368  java.util.regex.Matcher
  36:         11898         666288  java.beans.MethodDescriptor

● 增加jvm启动参数：-XX:+TraceClassLoading -XX:+TraceClassUnloading
● 查看项目的类加载日志

[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_InboxBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_RefundBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundLineBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundPushArgs from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_3_PushRefundLine from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_InboxBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_PushArgs from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_RefundBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundLineBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundPushArgs from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_3_PushRefundLine from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_InboxBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_RefundBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundLineBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]

xxx 03:29:32.045 [http-nio-8888-exec-7] INFO   xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
xxx 03:29:48.813 [http-nio-8888-exec-5] INFO   xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
xxx 03:29:56.844 [http-nio-8888-exec-2] INFO   xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
xxx 03:30:03.787 [http-nio-8888-exec-9] INFO   xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}

● 查看具体代码

public static String toSneakJSONString(Object object){
      SerializeConfig serializeConfig = new SerializeConfig();
      serializeConfig.setPropertyNamingStrategy(PropertyNamingStrategy.SnakeCase);
      return JSONObject.toJSONString(object, serializeConfig, SerializerFeature.DisableCircularReferenceDetect);
  }

● fastjson 官方建议

五、问题根源和解决方案

1.问题根源分析

SerializeConfig 默认会激活 asm，在序列化对象时会为对象生成代理类，然后通过执行代理进行序列化操作，通过这样优化来提高执行性能，但在应用不合理每次新创建 config 的时候就会导致大量生成代码类反而拖慢性能。反序列化时的 ParserConfig 也是同理。

在 jdk8 之前这些代理类会充满 Perm 区导致 FullGC，浪费点 CPU 也不会有大问题，但在 JDK8 中，这些类会大量创建直至充满物理机内存，操作系统检测到该进程是危险进程,出于自我保护机制,进而导致进程被系统杀掉。

2.解决方案

● 代码维度,将SerializeConfig缓存起来,或者使用SerializeConfig.globalInstance全局配置,避免请求维度动态生成类并加载到元空间
● 调整jvm启动参数,限制metaspace最大内存-XX:MaxMetaspaceSize,宁可FGC或者OOM也不要进程被杀

参考

fastjosn官方issues1

fastjson官方issues2

什么是元空间Metaspace

fastjson中用到的ASM

asm官网

本站文章资源均来源自网络，除非特别声明，否则均不代表站方观点，并仅供查阅，不作为任何参考依据！
如有侵权请及时跟我们联系，本站将及时删除！
如遇版权问题，请查看 本站版权声明
THE END
分享
二维码
海报
诡异的JVM堆外内存泄漏
● 报警详情: MEM usage above 90% (current value: 0.9731329333728482) ● 堆外metaspace...
<<上一篇
下一篇>>