Apache Storm (1.2.1) 传输异常断管

Question

我在最近升级的 Storm 1.2.1 版本中提交拓扑时出现以下错误。

69001 [pool-7-thread-1] WARN  o.a.s.u.NimbusClient - Ignoring exception while trying to get leader nimbus info from <ip>. will retry with a different seed host.
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
    at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:112) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:73) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.NimbusClient.<init>(NimbusClient.java:136) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:92) [storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:66) [storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:58) [storm-core-1.2.1.jar:1.2.1]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_79]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_79]
    at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
Caused by: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
    at org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:104) ~[storm-core-1.2.1.jar:1.2.1]
    ... 16 more

正在检查 nimbus.log 文件：

o.a.s.t.s.AbstractNonblockingServer$FrameBuffer Thread-9 [ERROR] Read a frame size of 16777216, which is bigger than the maximum allowable buffer size for ALL connections.

更新

已更改 nimbus.thrift.max_buffer_size：20000000 并且拓扑未提交，客户端和服务器中也没有 warn/error 日志。

在终止 nimbus 时，我收到管道异常，因为传输可能不完整。我不确定为什么这很慢。

这个问题的原因是什么？

Answer 1

设置storm.thrift.transport
"org.apache.storm.security.auth.plain.PlainSaslTransportPlugin" 在客户端配置和 storm.yml 文件中解决了这个问题。

我在 storm.yml 中使用了 SimpleTransportPlugin，在客户端配置中使用了 PlainSaslTransportPlugin，这导致了这个问题。

Apache Storm (1.2.1) 传输异常断管

Apache Storm (1.2.1) Transport exception Broken pipe

java

thrift

apache-storm