Reindex 调用中“include”参数中指定的字段数是否有限制
Is there a limit on the number of fields specified in the “include” param in Reindex call
这是我在执行 Reindex 调用时遇到的错误消息
[2020-03-13T16:08:39,042][WARN ][o.e.i.r.TransportReindexAction]
[ExportDatabase] giving up on search because it failed with a
non-retryable exception
org.elasticsearch.action.search.SearchPhaseExecutionException: all
shards failed at
org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:288)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:128)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:249)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.search.InitialSearchPhase.access0(InitialSearchPhase.java:48)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.search.InitialSearchPhase.lambda$onFailure(InitialSearchPhase.java:222)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.search.InitialSearchPhase.access[=11=]0(InitialSearchPhase.java:48)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.search.InitialSearchPhase.onFailure(InitialSearchPhase.java:222)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:527)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1095)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1188)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1172)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.search.SearchTransportService.onFailure(SearchTransportService.java:385)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.search.SearchService.onFailure(SearchService.java:341)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.search.SearchService.onResponse(SearchService.java:335)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.search.SearchService.onResponse(SearchService.java:329)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.search.SearchService.doRun(SearchService.java:1019)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
[elasticsearch-6.3.0.jar:6.3.0] at
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[?:1.8.0_212] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[?:1.8.0_212] at java.lang.Thread.run(Unknown Source) [?:1.8.0_212]
Caused by: org.elasticsearch.ElasticsearchException: Determinizing
automaton with 33501 states and 34977 transitions would result in more
than 10000 states. at
org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:658)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:126)
~[elasticsearch-6.3.0.jar:6.3.0] ... 26 more Caused by:
org.apache.lucene.util.automaton.TooComplexToDeterminizeException:
Determinizing automaton with 33501 states and 34977 transitions would
result in more than 10000 states. at
org.apache.lucene.util.automaton.Operations.determinize(Operations.java:746)
~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:69)
~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.CharacterRunAutomaton.(CharacterRunAutomaton.java:39)
~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.CharacterRunAutomaton.(CharacterRunAutomaton.java:28)
~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24] at org.elasticsearch.common.xcontent.support.XContentMapValues.filter(XContentMapValues.java:174)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.search.fetch.subphase.FetchSourceContext.getFilter(FetchSourceContext.java:234)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.search.lookup.SourceLookup.filter(SourceLookup.java:132)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.search.fetch.subphase.FetchSourceSubPhase.hitExecute(FetchSourceSubPhase.java:56)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:162)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:393)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:368)
~[elasticsearch-6.3.0.jar:6.3.0] at
org.elasticsearch.search.SearchService.onResponse(SearchService.java:333)
~[elasticsearch-6.3.0.jar:6.3.0] ... 9 more
所以根据我的分析，这是我正在执行的查询：
POST _reindex
{
"source":{
"index":"index_source",
"size":1000,
"query":
{
"terms":{
"ID":["7078"],
"boost":1.0
}
},
"_source":
{
"includes":here we have 1216 fields]
}
},
"dest":
{
"index":"index_dest"
}
}
在这里，如果我删除 include 参数中的 1216 个字段或将其减少到 500 左右，那么它就可以工作
所以在这方面需要帮助，有没有其他方法我们只能为这些特定字段重建索引，或者我做错了什么。
来自弹性搜索文档Mapping
1000 是索引中允许的最大字段数。
要增加限制，您可以更新 "index.mapping.total_fields.limit"
警告
The limit is in place to prevent mappings and searches from becoming
too large. Higher values can lead to performance degradations and
memory issues, especially in clusters with a high load or few
resources. If you increase this setting, we recommend you also
increase the indices.query.bool.max_clause_count setting, which limits
the maximum number of boolean clauses in a query.
这是我在执行 Reindex 调用时遇到的错误消息
[2020-03-13T16:08:39,042][WARN ][o.e.i.r.TransportReindexAction] [ExportDatabase] giving up on search because it failed with a non-retryable exception org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:288) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:128) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:249) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.access0(InitialSearchPhase.java:48) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.lambda$onFailure(InitialSearchPhase.java:222) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.access[=11=]0(InitialSearchPhase.java:48) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.onFailure(InitialSearchPhase.java:222) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:527) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1095) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1188) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1172) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.SearchTransportService.onFailure(SearchTransportService.java:385) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.onFailure(SearchService.java:341) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.onResponse(SearchService.java:335) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.onResponse(SearchService.java:329) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.doRun(SearchService.java:1019) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_212] at java.lang.Thread.run(Unknown Source) [?:1.8.0_212] Caused by: org.elasticsearch.ElasticsearchException: Determinizing automaton with 33501 states and 34977 transitions would result in more than 10000 states. at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:658) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:126) ~[elasticsearch-6.3.0.jar:6.3.0] ... 26 more Caused by: org.apache.lucene.util.automaton.TooComplexToDeterminizeException: Determinizing automaton with 33501 states and 34977 transitions would result in more than 10000 states. at org.apache.lucene.util.automaton.Operations.determinize(Operations.java:746) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:69) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.CharacterRunAutomaton.(CharacterRunAutomaton.java:39) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.CharacterRunAutomaton.(CharacterRunAutomaton.java:28) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.elasticsearch.common.xcontent.support.XContentMapValues.filter(XContentMapValues.java:174) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.fetch.subphase.FetchSourceContext.getFilter(FetchSourceContext.java:234) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.lookup.SourceLookup.filter(SourceLookup.java:132) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.fetch.subphase.FetchSourceSubPhase.hitExecute(FetchSourceSubPhase.java:56) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:393) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:368) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.onResponse(SearchService.java:333) ~[elasticsearch-6.3.0.jar:6.3.0] ... 9 more
所以根据我的分析，这是我正在执行的查询：
POST _reindex
{
"source":{
"index":"index_source",
"size":1000,
"query":
{
"terms":{
"ID":["7078"],
"boost":1.0
}
},
"_source":
{
"includes":here we have 1216 fields]
}
},
"dest":
{
"index":"index_dest"
}
}
在这里，如果我删除 include 参数中的 1216 个字段或将其减少到 500 左右，那么它就可以工作
所以在这方面需要帮助，有没有其他方法我们只能为这些特定字段重建索引，或者我做错了什么。
来自弹性搜索文档Mapping
1000 是索引中允许的最大字段数。 要增加限制，您可以更新 "index.mapping.total_fields.limit"
警告
The limit is in place to prevent mappings and searches from becoming too large. Higher values can lead to performance degradations and memory issues, especially in clusters with a high load or few resources. If you increase this setting, we recommend you also increase the indices.query.bool.max_clause_count setting, which limits the maximum number of boolean clauses in a query.