JVM 使用了比最大堆更多的 LOT

Question

我们遇到了一个 JVM 问题，它使用的数量远远超过 max -Xmx。

我们正在使用 -Xmx2048，它目前正在使用 17GB 的 OS 内存。

我意识到 JVM 可以使用超过最大堆的使用量，但我们的使用量超过 15GB，这看起来很疯狂。

顶部转储如下所示：

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                          
30477 root      20   0 22.683g 0.017t  18536 S   0.7 29.1  27:43.44 java -Xmx2048M -Xms1024M -XX:MaxPermSize=256M -XX:ReservedCodeCacheSize=128M  ....

（注意它使用了 0.017TB 或 17GB）。

我试过使用 jmap 进行调试：

sudo jmap -J-d64 -heap 30477

Attaching to process ID 30477, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.75-b04

using thread-local object allocation.
Parallel GC with 8 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 0
   MaxHeapFreeRatio = 100
   MaxHeapSize      = 2147483648 (2048.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 268435456 (256.0MB)
   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 316145664 (301.5MB)
   used     = 150828136 (143.8409194946289MB)
   free     = 165317528 (157.6590805053711MB)
   47.708431009827166% used
From Space:
   capacity = 20447232 (19.5MB)
   used     = 3406624 (3.248809814453125MB)
   free     = 17040608 (16.251190185546875MB)
   16.660563151041668% used
To Space:
   capacity = 21495808 (20.5MB)
   used     = 0 (0.0MB)
   free     = 21495808 (20.5MB)
   0.0% used
PS Old Generation
   capacity = 716177408 (683.0MB)
   used     = 516302952 (492.3848648071289MB)
   free     = 199874456 (190.6151351928711MB)
   72.09148825873044% used
PS Perm Generation
   capacity = 56098816 (53.5MB)
   used     = 55842840 (53.255882263183594MB)
   free     = 255976 (0.24411773681640625MB)
   99.54370516482915% used

15065 interned Strings occupying 1516808 bytes.

还有...

sudo jmap -J-d64 -permstat 30477

Attaching to process ID 30477, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.75-b04
finding class loader instances ..done.
computing per loader stat ..done.
please wait.. computing liveness...liveness analysis may be inaccurate ...
class_loader    classes bytes   parent_loader   alive?  type

<bootstrap> 1908    11793728      null      live    <internal>
0x0000000780b97590  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b99b28  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b96cb0  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780113ca0  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b97c90  1   3048    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780114458  1   3048      null      dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b892b0  1   3024    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b989a8  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780019cf0  7731    44085000    0x0000000780019d40  live    sun/misc/Launcher$AppClassLoader@0x0000000770239a20
0x0000000780113f10  1   3152    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780aa8918  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780114008  1   3040    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b7a8b8  1   3024    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780d236d0  1   3048    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780019d40  207 1327296   null      live    sun/misc/Launcher$ExtClassLoader@0x00000007701eb8b8
0x0000000780113e28  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x00000007801140b0  1   3064    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x000000078dc814f8  1   3024      null      dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b8abf8  1   3208    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b990a8  1   3048    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780114130  1   1904    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b98010  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x00000007801141b0  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780113620  27  86320   0x0000000780019cf0  live    org/apache/tapestry5/internal/plastic/PlasticClassLoader@0x000000077181b058
0x0000000780114418  1   3064    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b98d28  1   3048    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b89270  1   3024    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780113ed0  1   3024    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b97910  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b997a8  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b96930  1   3048    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780d36d38  1   3024    0x0000000780019d40  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b99ea8  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780bddff8  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780a63ca0  1   3024    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780114048  1   3048    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780113fc8  1   3024      null      dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780113de8  1   3072    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b7afb8  1   3040      null      dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780113e68  1   1904    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b7ac38  1   1880      null      dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780113d58  1   3064    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x00000007801140f0  1   3280    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b98628  1   3048    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x000000078088f900  0   0   0x0000000780019cf0  dead    java/util/ResourceBundle$RBClassLoader@0x00000007709cda40
0x0000000780b99428  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780d28ba0  1   3024    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b97030  1   3048    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780114170  1   3064    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x00000007801143d8  1   3048    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780114240  1   1880    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770
0x0000000780b8af60  1   3104    0x0000000780019cf0  dead    sun/reflect/DelegatingClassLoader@0x0000000770059770

total = 53  9921    57418080        N/A     alive=4, dead=49        N/A

该应用程序是一个 Play 应用程序，我知道它使用类加载器会做一些奇怪的事情，但我认为 -permstat 应该考虑到这一点，所以我很困惑。

有什么想法吗？

约翰

Answer 1

注意Heap Area（以Xmx为界）只是JVM进程的一部分。

基本上JVM占用的内存是以下总和：

堆（以 Xmx 为界）
PermGen（在 Java 7 及以下，以 MaxPermSize 为界）
线程堆栈：每个线程大约 512KB - 1024KB（取决于 JVM 和 OS）
原生缓冲区
JVM 代码
本机 data/code 通过 JNI 访问

可能还有更多内容，您可以通过谷歌搜索找到它:)

所以你必须检查所有这些区域以确保是什么在消耗内存。

我打赌你有简单的线程泄漏 - 数千个线程会占用大量内存（正如我所说，假设每个线程 1MB，10000 个线程对应 10GB 内存）。您可以简单地制作一个线程转储 (jstack -l > dumpfile.txt) 以查看它是否是一个问题。

当然也有可能是使用本地缓冲区或JNI的外部库，但可能性较小。

Answer 2

所以我们解决了这个问题，这只是开发人员的错误。感谢所有建议的帮助。

无论如何我都会回答这个问题，以防它在未来对任何人有所帮助。

问题只是没有关闭在使用 Class.getInputAsStream() 读取资源时打开的流。

混淆的产生是因为 JVM 堆的使用可以忽略不计，但每个未关闭的流都以某种方式留下了一些已分配但未释放的本机代码。

因为没有文件句柄泄漏（可能是因为我们每次都从主 JAR 文件中读取，它重复使用相同的句柄），并且因为 JVM 堆使用量很小，所以我们没有发现它，直到我们真正进行堆转储。

无论如何，这个故事的寓意是不要犯我犯过的错误，我假设如果 JVM 堆使用率很低，那么查看堆转储是没有意义的，因为在这种情况下会看到数千个仍然分配的对象给了我们一个线索。

谢谢大家！

JVM 使用了比最大堆更多的 LOT

JVM using a LOT more than the max Heap

java

memory

jvm