ELB 后面的 Apache CXF 中的 Broken Pipe Exception

Broken Pipe Exception in Apache CXF behind ELB

我有一个 restful 网络服务,它正在发送一个 zip 文件作为输出。 Restful Web 服务是使用 Apache CXF 开发的。该服务部署在 AWS 弹性负载均衡器后面。服务在 70% 的时间内正常运行。有时我们会收到下面提到的错误

Sending the response
04-Dec-2015 13:59:25.844 WARNING [http-nio-8080-exec-3] org.apache.cxf.phase.PhaseInterceptorChain.doDefaultLogging Interceptor for {http://webservices.heavywater.solutions/}HocrServicesImpl has thrown exception, unwinding now
 org.apache.cxf.interceptor.Fault: Could not send Message.
    at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
    at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:83)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
    at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:241)
    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153)
    at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:171)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1521)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1478)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393)
    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
    at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
    at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:295)
    at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:151)
    at org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStream.close(AbstractHTTPDestination.java:749)
    at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
    at org.apache.cxf.transport.http.AbstractHTTPDestination$BackChannelConduit.close(AbstractHTTPDestination.java:679)
    at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
    ... 34 more
Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:65)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:492)
    at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:127)
    at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
    at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:173)
    at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:139)
    at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:197)
    at org.apache.coyote.http11.InternalNioOutputBuffer.access[=10=]0(InternalNioOutputBuffer.java:41)
    at org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:320)
    at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:118)
    at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:256)
    at org.apache.coyote.Response.doWrite(Response.java:503)
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:388)
    ... 42 more

04-Dec-2015 13:59:25.948 WARNING [http-nio-8080-exec-3] org.apache.cxf.phase.PhaseInterceptorChain.doDefaultLogging Interceptor for {http://webservices.heavywater.solutions/}HocrServicesImpl has thrown exception, unwinding now
 org.apache.cxf.interceptor.Fault: Could not send Message.
    at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
    at org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:113)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:336)
    at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:83)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
    at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:241)
    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153)
    at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:171)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1521)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1478)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393)
    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
    at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
    at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:295)
    at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:151)
    at org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStream.close(AbstractHTTPDestination.java:749)
    at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
    at org.apache.cxf.transport.http.AbstractHTTPDestination$BackChannelConduit.close(AbstractHTTPDestination.java:679)
    at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
    ... 36 more
Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:65)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:492)
    at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:127)
    at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
    at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:173)
    at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:139)
    at org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:244)
    at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:189)
    at org.apache.coyote.http11.InternalNioOutputBuffer.access[=10=]0(InternalNioOutputBuffer.java:41)
    at org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:320)
    at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:116)
    at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:256)
    at org.apache.coyote.Response.doWrite(Response.java:503)
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:388)
    ... 44 more

这是发送响应的服务器端代码。

System.out.println("Sending the response");
File file = new File(folderPath+"/zipFile.zip");
ResponseBuilder builder = Response.ok((Object)file);
builder.header("Content-Type", "application/zip");
builder.header("Content-Disposition", "attachment; filename=\"zipFile.zip\"");
return builder.build();

客户端代码

RequestEntity entity = new InputStreamRequestEntity(stream);
            methodHeader.setName("content-type");
            methodHeader.setValue("application/octet-stream");
            methodHeader.setName("accept");
            methodHeader.setValue("application/zip");
            methodHeader.setName("content-length");
            methodHeader.setValue(fileSize);;
            mPost.addRequestHeader(methodHeader);
            mPost.setRequestEntity(entity);

            int statusCode = httpClient.executeMethod(mPost);

            System.out.println(statusCode);
            if(statusCode == 200){
                InputStream is = mPost.getResponseBodyAsStream();
                f = new File("/tmp/"+tiffFileName.replace(".tif", "_input.zip"));
                FileOutputStream fos = new FileOutputStream(f);
                try{
                    byte[] buf = new byte[1024];
                    int len = is.read(buf);
                    while(len>0){
                        fos.write(buf, 0, len);
                        len = is.read(buf);
                    }           
                }finally{
                    if(fos != null)
                        fos.close();
                }
            }else if(statusCode == 403){
                System.out.println("Invalid username/password");
            }else{
                System.out.println(mPost.getResponseBodyAsString());
            }


        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }finally{
            if(mPost !=null){
                mPost.releaseConnection();
            }
        }

默认情况下,弹性负载均衡器空闲超时为 60 秒。如果您的 Web 服务请求花费的时间超过 60 秒,那么连接将超时并且您将得到 broken pipe 异常。要解决此问题,请在 ELB

中进行以下更改
  1. 在 ELB 设置 > 描述 > 连接设置下增加空闲超时
  2. 增加 tomcat 服务器配置文件 (server.xml) 中的连接超时

<Connector port="8080" redirectPort="8443" connectionTimeout="900000" protocol="HTTP/1.1"/>

希望这会解决问题。

我也面临着这个管道破裂的问题,而且是一个棘手的问题。 问题是间歇性的,仅发生在 AWS 环境中,并且在 on-premise 和本地服务器中运行良好。

交换 header 中设置的内容非常大时会出现问题。 通过在交换 "property" 而不是 "header" 中设置内容来解决它。 它在 non-Cloud 环境中工作的原因是 Tomcat header max size(server.max-http-header-size) 显式增加,但未选择相同的 Web 服务器设置在云端。

希望对遇到类似问题的其他人有所帮助。