Tomcat 响应过程中有 1000 多个线程阻塞

Question

问题第一：

我的服务响应时间随着并发增加而增加。我在 tomcat 响应客户端的步骤中发现了 1000 多个线程块。但我不知道如何解决。

背景：

我的服务是一个 Spring 引导项目，其中嵌入了 tomcat。 (tomcat-embed-core-8.5.23)

我的实验是使用 JMeter 作为客户端，创建 2000 个线程向我的服务发送请求 10 轮（总共 20000 个请求）。在我的服务之后，有一个存根服务可以在每个请求延迟 2 秒内响应。（即 JMeter（2000 线程）=> 我的服务 => 存根服务器）

实验数据

在我的服务中，我设置了“server.tomcat.max-threads=2000”。所以有 2000 多个线程名为“http-nio-8080*”
结果是，最大响应时间 >30 秒。这是从 CLI 复制的，

summary = 20000 in 00:01:23 = 241.7/s Avg: 6890 Min: 2288 Max: 31158 Err: 0 (0.00%)

最小响应时间 (2288ms) 符合预期（因为存根服务器中有 2000ms 的成本）。但是最大响应时间 (31158 ms) 太长了。我分析了jstack日志。有 1121 个线程阻塞在同一个锁上。（附上其中一个堆栈内容，我认为是My service响应JMeter的时间点，可惜有一个SynchronizedStack阻塞了1000+线程）

我有证据表明锁是响应时间的真正杀手。在9秒后的另一个jstack日志中，仍然有1058个线程锁在同一个锁上。

"http-nio-8080-exec-3988" #4038 daemon prio=5 os_prio=0 tid=0x00007fdec045c800 nid=0x1050 waiting for monitor entry [0x00007fdb6c94e000]
   java.lang.Thread.State: BLOCKED (on object monitor)
                at org.apache.tomcat.util.collections.SynchronizedStack.pop(SynchronizedStack.java:75)
                - waiting to lock <0x00000000dac7afd0> (a org.apache.tomcat.util.collections.SynchronizedStack)
                at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:89)
                at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157)
                at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1267)
                at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:670)
                at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:607)
                at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:597)
                at org.apache.coyote.http11.Http11OutputBuffer.flushBuffer(Http11OutputBuffer.java:581)
                at org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:272)
                at org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1560)
                at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:283)
                at org.apache.coyote.Response.action(Response.java:173)
                at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:317)
                at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:284)
                at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)
                at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
                at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
                - locked <0x00000000fe000000> (a java.io.OutputStreamWriter)
                at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
                at org.springframework.util.StreamUtils.copy(StreamUtils.java:119)
                at org.springframework.http.converter.StringHttpMessageConverter.writeInternal(StringHttpMessageConverter.java:106)
                at org.springframework.http.converter.StringHttpMessageConverter.writeInternal(StringHttpMessageConverter.java:41)
                at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:227)
                at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:247)
                at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:174)
                at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:81)
                at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:113)
                at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
                at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
                at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
                at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
                at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
                at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
                at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
                at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
                at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
                at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
                at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
                at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
                at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
                - locked <0x00000000e1a59188> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
                at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
                - <0x00000000df55e210> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Answer 1

您错误地允许 tomcat 过度使用。

首先，将 tomcat 执行程序线程限制在最多 100 或 200 个左右。实际上，任何 jvm 中的线程都不应该超过 200 个，除非你真的知道自己在做什么并且有 99% 的线程在等待（没有阻塞监视器条目）。

然后，限制 tcp SYN 队列长度（serversocket 接受队列）以确保在连接过多时不会接受所有 jmeter 连接。

从严格限制线程开始，只要您的网络带宽和 CPU 都空闲，就增加线程数量，即便如此，只要响应时间不增加。

Tomcat 响应过程中有 1000 多个线程阻塞

1000+ threads blocks on Tomcat response process

multithreading

spring-boot

embedded-tomcat-8