AWS Elasticsearch Service 节点尝试连接到未知目的地

Question

我有一个 AWS Elasticsearch Service 集群运行以下规格：

版本 7.9.1，服务软件版本 R20210426-P2
3 个节点，无专用主节点
实例类型：t3.medium.elasticsearch

我通过 Cloudwatch 启用了应用程序日志，并注意到每秒都会记录以下消息：

[2021-08-19T04:41:42,118][WARN ][r.suppressed] [NODE_ID] path: __PATH__ params: {redirectEndpoint=metrics}
java.net.ConnectException: Connection refused (Connection refused)

完成堆栈跟踪：

[2021-08-17T09:30:42,333][WARN ][r.suppressed             ] [NODE_ID] path: __PATH__ params: {redirectEndpoint=metrics}
java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[?:?]
    at java.net.Socket.connect(Socket.java:609) ~[?:?]
    at org.conscrypt.AbstractConscryptSocket.connect(AbstractConscryptSocket.java:155) ~[conscrypt-openjdk-uber-1.4.0.jar:1.4.0]
    at sun.net.NetworkClient.doConnect(NetworkClient.java:177) ~[?:?]
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:474) ~[?:?]
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:569) ~[?:?]
    at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265) ~[?:?]
    at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372) ~[?:?]
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.run(HttpURLConnection.java:1071) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.run(HttpURLConnection.java:1069) ~[?:?]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
    at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:795) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1068) ~[?:?]
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.run(HttpURLConnection.java:1507) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.run(HttpURLConnection.java:1505) ~[?:?]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
    at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:795) ~[?:?]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1504) ~[?:?]
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) ~[?:?]
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334) ~[?:?]
__AMAZON_INTERNAL__
    at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:94) ~[elasticsearch-7.9.1.jar:7.9.1]
__AMAZON_INTERNAL__
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:236) [elasticsearch-7.9.1.jar:7.9.1]
    at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:318) [elasticsearch-7.9.1.jar:7.9.1]
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:176) [elasticsearch-7.9.1.jar:7.9.1]
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
    at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:264) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
__AMAZON_INTERNAL__
    at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
__AMAZON_INTERNAL__
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
    at org.eclipse.jetty.server.Server.handle(Server.java:370) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
    at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) [jetty-http-8.1.12.v20130726.jar:8.1.12.v20130726]
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) [jetty-http-8.1.12.v20130726.jar:8.1.12.v20130726]
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) [jetty-server-8.1.12.v20130726.jar:8.1.12.v20130726]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) [jetty-io-8.1.12.v20130726.jar:8.1.12.v20130726]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:52) [jetty-io-8.1.12.v20130726.jar:8.1.12.v20130726]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) [jetty-util-8.1.12.v20130726.jar:8.1.12.v20130726]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.run(QueuedThreadPool.java:543) [jetty-util-8.1.12.v20130726.jar:8.1.12.v20130726]
    at java.lang.Thread.run(Thread.java:834) [?:?]

节点似乎在尝试连接到服务器。起初我以为一个节点无法到达集群中的另一个节点，但即使我的开发环境只有一个节点，每秒也会记录此消息。

不知道是AWS ES错误/我自己配置错误还是OpenDistro的一般问题

如果有人可以帮助我理解此错误消息，我将不胜感激 and/or 可以让我走上正确的轨道来解决此问题。 感谢任何帮助。

提前致谢！

Answer 1

好吧，有趣的是，由于 jvm 压力高，我将实例类型升级到 m6g.large.elasticsearch 后，这个问题就停止了。然而，我的开发和暂存单节点集群（使用 t3.small.elasticsearch 实例）仍在记录此异常。

我不是 100% 确定升级您的实例类型是否是解决此问题的关键。现在我会接受我自己的回答。如果我有新的见解，我会告诉你的。

AWS Elasticsearch Service 节点尝试连接到未知目的地

AWS Elasticsearch Service node tries to connect to unknown destination

amazon-web-services

elasticsearch

elasticsearch-opendistro