tomcat 类加载器性能问题

Question

我们的基准测试发现了 Apache Tomcat 7.0.59 中的一个漏洞。当服务器达到其性能极限时，其大部分线程被 ClassLoader 锁定。

阻塞线程的堆栈跟踪类似于此示例：

"http-bio-4504-exec-500" Id=2335 BLOCKED on java.util.jar.JarFile@464f9f8 owned by "[1432628598653] POST /services/signin HTTP/1.0" Id=1990
    at java.util.zip.ZipFile.getEntry(ZipFile.java:304)
    -  blocked on java.util.jar.JarFile@464f9f8
    at java.util.jar.JarFile.getEntry(JarFile.java:226)
    at java.util.jar.JarFile.getJarEntry(JarFile.java:209)
    at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:840)
    at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:818)
    at sun.misc.URLClassPath.next(URLClassPath.java:226)
    at sun.misc.URLClassPath.hasMoreElements(URLClassPath.java:236)
    at java.net.URLClassLoader.run(URLClassLoader.java:583)
    at java.net.URLClassLoader.run(URLClassLoader.java:581)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.next(URLClassLoader.java:580)
    at java.net.URLClassLoader.hasMoreElements(URLClassLoader.java:605)
    at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
    at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
    at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
    at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
    at com.sun.xml.ws.util.ServiceFinder$LazyIterator.hasNext(ServiceFinder.java:443)
    at com.sun.xml.ws.util.ServiceFinder$CompositeIterator.hasNext(ServiceFinder.java:390)
    at com.sun.xml.ws.api.message.saaj.SAAJFactory.getMessageFactory(SAAJFactory.java:96)
    at com.sun.xml.ws.api.SOAPVersion.getMessageFactory(SOAPVersion.java:221)
    at com.sun.xml.ws.api.message.saaj.SAAJFactory.readAsSOAPMessage(SAAJFactory.java:275)
    at com.sun.xml.ws.api.message.saaj.SAAJFactory.readAsSAAJ(SAAJFactory.java:205)
    at com.sun.xml.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:194)
    at com.sun.xml.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:199)
    at com.sun.xml.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:160)
    at com.sun.xml.ws.handler.SOAPMessageContextImpl.getMessage(SOAPMessageContextImpl.java:86)

阻塞线程运行在同一个地方

"[1432628598653] POST /services/signin HTTP/1.0" Id=1990 RUNNABLE
    at java.util.zip.ZipFile.getEntry(ZipFile.java:304)
    -  locked java.util.jar.JarFile@464f9f8
    at java.util.jar.JarFile.getEntry(JarFile.java:226)
    at java.util.jar.JarFile.getJarEntry(JarFile.java:209)
    at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:840)
    at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:818)
    at sun.misc.URLClassPath.next(URLClassPath.java:226)
    at sun.misc.URLClassPath.hasMoreElements(URLClassPath.java:236)
    at java.net.URLClassLoader.run(URLClassLoader.java:583)
    at java.net.URLClassLoader.run(URLClassLoader.java:581)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.next(URLClassLoader.java:580)
    at java.net.URLClassLoader.hasMoreElements(URLClassLoader.java:605)
    at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
    at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
    at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
    at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
    at com.sun.xml.ws.util.ServiceFinder$LazyIterator.hasNext(ServiceFinder.java:443)
    at com.sun.xml.ws.util.ServiceFinder$CompositeIterator.hasNext(ServiceFinder.java:390)
    at com.sun.xml.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:190)
    at com.sun.xml.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:199)
    at com.sun.xml.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:160)
    at com.sun.xml.ws.handler.SOAPMessageContextImpl.getMessage(SOAPMessageContextImpl.java:86)

我们的应用程序是否以错误的方式执行某些操作？是否有避免在类加载器中阻塞的解决方法？

Answer 1

问题已通过以下 hack 完全解决：

我在 JVM 参数中添加了 -Djaxp.debug=true。
然后我在日志中找到了 JAX-WS 正在寻找的所有 classes。
我为每个提到的 class 添加了系统属性，它使 ServiceFinder 忽略了在 class 路径中搜索 service.properties 文件。

我最终的 JVM args 看起来像：

-Djavax.xml.soap.MetaFactory=com.sun.xml.internal.messaging.saaj.soap.SAAJMetaFactoryImpl
-Djavax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.DatatypeFactoryImpl
-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl
-Djavax.xml.stream.XMLInputFactory=com.ctc.wstx.stax.WstxInputFactory
-Djavax.xml.stream.XMLOutputFactory=com.ctc.wstx.stax.WstxOutputFactory
-Djavax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl

tomcat 类加载器性能问题

tomcat classloader performance issue

multithreading

tomcat

web-services

classloader

saaj