Zuul Forwarding Exception caused by org.apache.http.NoHttpResponseException: 目标服务器响应失败

Zuul Forwarding Exception caused by org.apache.http.NoHttpResponseException: The target server failed to respond

你好 Stack Overflow 社区,

自从我问了一个问题以来已经有一段时间了,所以如果我忘记了一些信息,请告诉我,我会很乐意添加。

我们最近发现了我们的客户端-服务器中发生的一些问题。我们是 运行 一个 Spring 服务器,它处理客户端请求并通过 Thymeleaf 呈现 HTML。我们使用 Zuul 将我们的许多客户端路由重定向到单个 Spring 端点,该端点提供单个捆绑包,这样无论用户在哪里刷新,他们都会收到我们希望他们获得的捆绑包。这似乎在大多数时间都有效,但大约每分钟一次(每分钟数千次)我们会得到一个 zuul 转发异常,如下所示:

[2m2020-11-30 02:26:51.023[0;39m [33m WARN[0;39m [35m50[0;39m [2m---[0;39m [33m[trace=,span=][0;39m [2m[nio-8080-exec-2][0;39m [36mo.s.c.n.z.filters.post.SendErrorFilter  [0;39m 
[2m:[0;39m Error during filteringcom.netflix.zuul.exception.ZuulException: Forwarding error
    at 
org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.handleException(SimpleHostRoutingFilter.java:261) ~[spring-cloud-netflix-zuul-2.2.1.RELEASE.jar:2.2.1.RELEASE]
    at 
org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.run(SimpleHostRoutingFilter.java:241) ~[spring-cloud-netflix-zuul-2.2.1.RELEASE.jar:2.2.1.RELEASE]
    at 
com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:117) ~[zuul-core-1.3.1.jar:1.3.1]
    at 
com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:193) ~[zuul-core-1.3.1.jar:1.3.1]
  
at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:157) ~[zuul-core-1.3.1.jar:1.3.1]
  at 
com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:118) ~[zuul-core-1.3.1.jar:1.3.1]
  at 
com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96) ~[zuul-core-1.3.1.jar:1.3.1]
 at 
com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116) ~[zuul-core-1.3.1.jar:1.3.1]
 at 
com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81) ~[zuul-core-1.3.1.jar:1.3.1]
    at 
org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:166) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at 
org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequest(ZuulController.java:45) [spring-cloud-netflix-zuul-2.2.1.RELEASE.jar:2.2.1.RELEASE]
    at 
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:52) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
  at 
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
  at 
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at 
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at 
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:626) [tomcat-embed-core-9.0.37.jar:4.0.FR]
  at 
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) [tomcat-embed-core-9.0.37.jar:4.0.FR]
  at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.37.jar:9.0.37]
  at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:64) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
 at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
   at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
   at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
  at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
   at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:109) [spring-boot-actuator-2.2.7.RELEASE.jar:2.2.7.RELEASE]
 at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
   at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
 at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
   at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.cloudfoundry.router.ClientCertificateMapper.doFilter(ClientCertificateMapper.java:79) [client_certificate_mapper-1.11.0_RELEASE.jar:na]
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.37.jar:9.0.37]
  at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.37.jar:9.0.37]
   at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat-embed-core-9.0.37.jar:9.0.37]
   at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.37.jar:9.0.37]
    at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.37.jar:9.0.37]
 at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.37.jar:9.0.37]
 at 
org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) [tomcat-embed-core-9.0.37.jar:9.0.37]
  at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.37.jar:9.0.37]
  at 
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) [tomcat-embed-core-9.0.37.jar:9.0.37]
   at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.37.jar:9.0.37]
 at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-9.0.37.jar:9.0.37]
  at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589) [tomcat-embed-core-9.0.37.jar:9.0.37]
  at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.37.jar:9.0.37]
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_242]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_242]
  at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.37.jar:9.0.37]
   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]


Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond
   at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) ~[httpclient-4.5.12.jar:4.5.12]
  at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.12.jar:4.5.12]
   at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.13.jar:4.4.13]
  at 
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.13.jar:4.4.13]
   at 
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.12.jar:4.5.12]
    at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.13.jar:4.4.13]
 at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.13.jar:4.4.13]
   at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.12.jar:4.5.12]
 at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.12.jar:4.5.12]
 at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.12.jar:4.5.12]
 at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.12.jar:4.5.12]
  at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) ~[httpclient-4.5.12.jar:4.5.12]
  at 
org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.forwardRequest(SimpleHostRoutingFilter.java:422) ~[spring-cloud-netflix-zuul-2.2.1.RELEASE.jar:2.2.1.RELEASE]
 at 
org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.forward(SimpleHostRoutingFilter.java:341) ~[spring-cloud-netflix-zuul-2.2.1.RELEASE.jar:2.2.1.RELEASE]
    at 
org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.run(SimpleHostRoutingFilter.java:236) ~[spring-cloud-netflix-zuul-2.2.1.RELEASE.jar:2.2.1.RELEASE]
    ... 61 common frames omitted


如您所见,我们使用的是 HttpClient 4.5.12。我在此 中看到 4.4 中出现了类似的问题,但我们升级后问题仍然存在。在同一个 post @daimarom 评论说,也许 PoolingHttpClientConnectionManager 可能会导致问题。这就是我们配置池管理器的方式:

 @Bean
 public PoolingHttpClientConnectionManager connectionManager() {
        
     PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
     connectionManager.setValidateAfterInactivity(5000);
     connectionManager.setDefaultMaxPerRoute(40);
     connectionManager.setMaxTotal(15);
     return connectionManager;
 }

 @Bean
 public CloseableHttpClient httpClient() throws Exception {
     return HttpClients.custom()
         .disableAutomaticRetries()
         .setSSLContext(sslContext())
         .setSSLSocketFactory(socketFactory())
         .setConnectionManager(connectionManager())
         .setSSLHostnameVerifier(new NoopHostnameVerifier())
         .build();
 }

尝试使用连接管理器参数没有帮助。

我们有一个 Zuul 过滤器,但我们的路由完全基于配置以排除任何逻辑错误。目前我们的 Zuul 配置如下所示:

zuul:
  sensitive-headers: 
  host.socket-timeout-millis: 60000
  routes:
    api:
      strip-prefix: false
      path: /api/**
      url: "${validation.urls.piserver}"
    extract:
      strip-prefix: true
      path: /web2/extract/**
      url: "${validation.urls.piclient}/validate"
    legal:
      strip-prefix: true
      path: /web2/legal/**
      url: "${validation.urls.piclient}/validate"
    capture-vin:
      strip-prefix: true
      path: /web2/capture-instructions/vin/**
      url: "${validation.urls.piclient}/validate"
    recapture-vin:
      strip-prefix: true
      path: /web2/recapture-instructions/vin/**
      url: "${validation.urls.piclient}/validate"
    capture-odometer:
      strip-prefix: true
      path: /web2/capture-instructions/odometer/**
      url: "${validation.urls.piclient}/validate"
    recapture-odometer:
      strip-prefix: true
      path: /web2/recapture-instructions/odometer/**
      url: "${validation.urls.piclient}/validate"
    capture-rear:
      strip-prefix: true
      path: /web2/capture-instructions/rear/**
      url: "${validation.urls.piclient}/validate"
    recapture-rear:
      strip-prefix: true
      path: /web2/recapture-instructions/rear/**
      url: "${validation.urls.piclient}/validate"
    capture-rear-passenger:
      strip-prefix: true
      path: /web2/capture-instructions/rear-passenger/**
      url: "${validation.urls.piclient}/validate"
    recapture-rear-passenger:
      strip-prefix: true
      path: /web2/recapture-instructions/rear-passenger/**
      url: "${validation.urls.piclient}/validate"
    capture-front-passenger:
      strip-prefix: true
      path: /web2/capture-instructions/front-passenger/**
      url: "${validation.urls.piclient}/validate"
    recapture-front-passenger:
      strip-prefix: true
      path: /web2/recapture-instructions/front-passenger/**
      url: "${validation.urls.piclient}/validate"
    capture-front-driver:
      strip-prefix: true
      path: /web2/capture-instructions/front-driver/**
      url: "${validation.urls.piclient}/validate"
    recapture-front-driver:
      strip-prefix: true
      path: /web2/recapture-instructions/front-driver/**
      url: "${validation.urls.piclient}/validate"
    capture-rear-driver:
      strip-prefix: true
      path: /web2/capture-instructions/rear-driver/**
      url: "${validation.urls.piclient}/validate"
    recapture-rear-driver:
      strip-prefix: true
      path: /web2/recapture-instructions/rear-driver/**
      url: "${validation.urls.piclient}/validate"
    capture-damage-left:
      strip-prefix: true
      path: /web2/capture-instructions/damage-left/**
      url: "${validation.urls.piclient}/validate"
    recapture-damage-left:
      strip-prefix: true
      path: /web2/recapture-instructions/damage-left/**
      url: "${validation.urls.piclient}/validate"
    capture-damage-center:
      strip-prefix: true
      path: /web2/damage-capture-instructions/damage-center/**
      url: "${validation.urls.piclient}/validate"
    recapture-damage-center:
      strip-prefix: true
      path: /web2/damage-recapture-instructions/damage-center/**
      url: "${validation.urls.piclient}/validate"
    capture-damage-right:
      strip-prefix: true
      path: /web2/damage-capture-instructions/damage-right/**
      url: "${validation.urls.piclient}/validate"
    recapture-damage-right:
      strip-prefix: true
      path: /web2/damage-recapture-instructions/damage-right/**
      url: "${validation.urls.piclient}/validate"
    capture-damages:
      strip-prefix: true
      path: /web2/capture-instructions/damages/**
      url: "${validation.urls.piclient}/validate"
    recapture-damages:
      strip-prefix: true
      path: /web2/recapture-instructions/damages/**
      url: "${validation.urls.piclient}/validate"
    uploading:
      strip-prefix: true
      path: /web2/uploading/**
      url: "${validation.urls.piclient}/validate"
    submit:
      strip-prefix: true
      path: /submit/**
      url: "${validation.urls.piclient}/validate"
    intake-processing:
      strip-prefix: true
      path: /intake-processing/**
      url: "${validation.urls.piclient}/validate"
    already-submitted:
      strip-prefix: true
      path: /already-submitted/**
      url: "${validation.urls.piclient}/validate"
    submit-v1:
      strip-prefix: true
      path: /submit
      url: "${validation.urls.piclient}/validate"
    intake-processing-v1:
      strip-prefix: true
      path: /intake-processing
      url: "${validation.urls.piclient}/validate"
    already-submitted-v1:
      strip-prefix: true
      path: /already-submitted
      url: "${validation.urls.piclient}/validate"

当我们在当前上下文中简单地重置请求 URI 时,这在路由过滤器中更加简洁,但正如我之前提到的,现在这都是基于配置的。

我们所有的依赖都在这里定义:

dependencies {

    compileOnly 'org.projectlombok:lombok'

    annotationProcessor 'org.projectlombok:lombok'
    annotationProcessor 'org.springframework.boot:spring-boot-configuration-processor'

    implementation 'org.yaml:snakeyaml:1.26'
    implementation 'io.github.openfeign:feign-httpclient:10.7.4'
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'org.springframework.boot:spring-boot-starter-thymeleaf'
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'org.springframework.cloud:spring-cloud-starter-openfeign'
    implementation 'org.springframework.cloud:spring-cloud-starter-netflix-zuul'

    implementation 'org.apache.tomcat.embed:tomcat-embed-core:9.0.37'
    implementation 'org.apache.tomcat.embed:tomcat-embed-el:9.0.37'
    implementation 'org.apache.tomcat.embed:tomcat-embed-websocket:9.0.37'

    implementation 'org.springframework.cloud:spring-cloud-cloudfoundry-connector'

    testImplementation 'org.springframework.security:spring-security-test'
    testImplementation('org.springframework.boot:spring-boot-starter-test') {
        exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'
    }
}

我觉得我在这里遗漏了一些非常明显的东西,但我们对 Zuul 非常有经验的经理最近离开了。如果我可以在此处添加任何其他信息,请告诉我。提前感谢您的帮助。

[更新] 我最近尝试将整个池管理器从配置中移除。问题一直存在。现在 HttpClient 的配置如下所示:

@Bean
public CloseableHttpClient httpClient() throws Exception {
    return HttpClients.custom()
        .disableAutomaticRetries()
        .setSSLContext(sslContext())
        .setSSLSocketFactory(socketFactory())
        // .setConnectionManager(connectionManager())
        .setSSLHostnameVerifier(new NoopHostnameVerifier())
        .build();
}

此外,我们尝试了使用和不使用显式定义的 spring 安全配置链。没有任何帮助。

好的,@brijesh 在评论中提到了一个我尚未尝试的潜在解决方案,但我最终部署了另一个“修复”更改。这更像是从很可能仍在发生的问题中恢复过来。我只是通过注释掉以下行来启用自动重试:

@Bean
public CloseableHttpClient httpClient() throws Exception {
    return HttpClients.custom()
        // .disableAutomaticRetries()
        .setSSLContext(sslContext())
        .setSSLSocketFactory(socketFactory())
        .setConnectionManager(connectionManager())
        .setSSLHostnameVerifier(new NoopHostnameVerifier())
        .build();
}

对于没有明确配置其 HttpClient 设置的任何人,这应该不是问题,因为默认设置是重试 3 次。

对于确实配置了这些设置并且连接管理器有问题的人,我将很快尝试@brijesh 的方法(禁用自动重试)以查看解决方案是否解决了根本原因。完成后我会在这里更新答案。