弹性搜索边缘 ngram 未返回所有预期结果

Question

我很难找到弹性搜索查询的意外结果。将以下文档编入弹性搜索索引。

{
"group": "J00-I99", codes: [
   { "id": "J15", "description": "hello world" },
   { "id": "J15.0", "description": "test one world" },
   { "id": "J15.1", "description": "test two world J15.0" },
   { "id": "J15.2", "description": "test two three world J15" },
   { "id": "J15.3", "description": "hello world J18 " },
    ............................ // Similar records here
   { "id": "J15.9", "description": "hello world new" },
   { "id": "J16.0", "description": "new description" }
]
}

我的目标是实现自动完成功能，为此我使用了 n-gram 方法。我不想使用完整的建议方法。

目前我遇到两个问题：

搜索查询（ID 和描述字段）：J15

预期结果：以上所有结果，包括J15 实际结果：只得到很少的结果（J15.0、J15.1、J15.8）

搜索查询（id 和 description 字段）：测试两个

预期结果：

{ "id": "J15.1", "description": "test two world J15.0" },
{ "id": "J15.2", "description": "test two three world J15" },

实际结果：

   { "id": "J15.0", "description": "test one world" },
   { "id": "J15.1", "description": "test two world J15.0" },
   { "id": "J15.2", "description": "test two three world J15" },

然后映射就这样完成了

           {

                settings: {
                    number_of_shards: 1,
                    analysis: {
                        filter: {
                            ngram_filter: {
                                type: 'edge_ngram',
                                min_gram: 2,
                                max_gram: 20
                            }
                        },
                        analyzer: {
                            ngram_analyzer: {
                                type: 'custom',
                                tokenizer: 'standard',
                                filter: [
                                    'lowercase', 'ngram_filter'
                                ]
                            }
                        }
                    }
                },
                mappings: {
                    properties: {
                        group: {
                            type: 'text'
                        },
                        codes: {
                            type: 'nested',
                            properties: {
                                id: {
                                    type: 'text',
                                    analyzer: 'ngram_analyzer',
                                    search_analyzer: 'standard'
                                },
                                description: {
                                    type: 'text',
                                    analyzer: 'ngram_analyzer',
                                    search_analyzer: 'standard'
                                }
                            }
                        }
                    }
                }
            }

搜索查询：

GET myindex/_search
{
  "_source": {
    "excludes": [
      "codes"
    ]
  },
  "query": {
    "nested": {
      "path": "codes",
      "query": {
        "bool": {
          "should": [
            {
              "match": {
                "codes.description": "J15"
              }
            },
            {
              "match": {
                "codes.id": "J15"
              }
            }
          ]
        }
      },
      "inner_hits": {}
    }
  }
}

注意：文档索引会很大。这里只提到示例数据。

对于第二个问题，我可以像下面那样使用 multi_match 和 AND 运算符吗？

GET myindex/_search
{
  "_source": {
    "excludes": [
      "codes"
    ]
  },
  "query": {
    "nested": {
      "path": "codes",
      "query": {
        "bool": {
          "should": [
            {
              "multi_match": {
                    "query": "J15",
                    "fields": ["codes.id", "codes.description"],
                    "operator": and
                }
            }
          ]
        }
      },
      "inner_hits": {}
    }
  }
}

任何帮助将不胜感激，因为我很难解决这个问题。

Answer 1

添加具有索引映射、搜索查询和搜索结果的工作示例

索引映射：

{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "my_tokenizer"
        }
      },
      "tokenizer": {
        "my_tokenizer": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 20,
          "token_chars": [
            "letter",
            "digit"
          ]
        }
      }
    },
    "max_ngram_diff": 50
  },
  "mappings": {
    "properties": {
      "group": {
        "type": "text"
      },
      "codes": {
        "type": "nested",
        "properties": {
          "id": {
            "type": "text",
            "analyzer": "my_analyzer"
          }
        }
      }
    }
  }
}

索引数据：

{
    "group": "J00-I99", 
    "codes": [
        {
            "id": "J15",
            "description": "hello world"
        },
        {
            "id": "J15.0",
            "description": "test one world"
        },
        {
            "id": "J15.1",
            "description": "test two world J15.0"
        },
        {
            "id": "J15.2",
            "description": "test two three world J15"
        },
        {
            "id": "J15.3",
            "description": "hello world J18 "
        },
        {
            "id": "J15.9",
            "description": "hello world new"
        },
        {
            "id": "J16.0",
            "description": "new description"
        }
    ]
}

搜索查询：

{
    "_source": {
        "excludes": [
            "codes"
        ]
    },
    "query": {
        "nested": {
            "path": "codes",
            "query": {
                "bool": {
                    "should": [
                        {
                            "match": {
                                "codes.description": "J15"
                            }
                        },
                        {
                            "match": {
                                "codes.id": "J15"
                            }
                        }
                    ],
                    "must": {
                        "multi_match": {
                            "query": "test two",
                            "fields": [
                                "codes.id",
                                "codes.description"
                            ],
                            "type": "phrase"
                        }
                    }
                }
            },
            "inner_hits": {}
        }
    }
}

搜索结果：

"inner_hits": {
          "codes": {
            "hits": {
              "total": {
                "value": 2,
                "relation": "eq"
              },
              "max_score": 3.2227304,
              "hits": [
                {
                  "_index": "stof_64170045",
                  "_type": "_doc",
                  "_id": "1",
                  "_nested": {
                    "field": "codes",
                    "offset": 3
                  },
                  "_score": 3.2227304,
                  "_source": {
                    "id": "J15.2",
                    "description": "test two three world J15"
                  }
                },
                {
                  "_index": "stof_64170045",
                  "_type": "_doc",
                  "_id": "1",
                  "_nested": {
                    "field": "codes",
                    "offset": 2
                  },
                  "_score": 2.0622847,
                  "_source": {
                    "id": "J15.1",
                    "description": "test two world J15.0"
                  }
                }
              ]
            }
          }
        }
      }

Answer 2

问题是默认情况下 inner_hits returns 只有 3 个匹配文档，如 this official doc、

中所述

size

The maximum number of hits to return per inner_hits. By default the top three matching hits are returned.

只需在 inner_hits 中添加 size 参数即可获得所有搜索结果。

  "inner_hits": {
                "size": 10 // note this
            }

在您的示例数据上尝试了此操作并查看了您的第一个查询的搜索结果，该查询仅返回 3 个搜索结果

第一次查询搜索结果

   "hits": [
                                {
                                    "_index": "myindexedge64170045",
                                    "_type": "_doc",
                                    "_id": "1",
                                    "_nested": {
                                        "field": "codes",
                                        "offset": 2
                                    },
                                    "_score": 1.8687118,
                                    "_source": {
                                        "id": "J15.1",
                                        "description": "test two world J15.0"
                                    }
                                },
                                {
                                    "_index": "myindexedge64170045",
                                    "_type": "_doc",
                                    "_id": "1",
                                    "_nested": {
                                        "field": "codes",
                                        "offset": 3
                                    },
                                    "_score": 1.7934312,
                                    "_source": {
                                        "id": "J15.2",
                                        "description": "test two three world J15"
                                    }
                                },
                                {
                                    "_index": "myindexedge64170045",
                                    "_type": "_doc",
                                    "_id": "1",
                                    "_nested": {
                                        "field": "codes",
                                        "offset": 0
                                    },
                                    "_score": 0.29618382,
                                    "_source": {
                                        "id": "J15",
                                        "description": "hello world"
                                    }
                                },
                                {
                                    "_index": "myindexedge64170045",
                                    "_type": "_doc",
                                    "_id": "1",
                                    "_nested": {
                                        "field": "codes",
                                        "offset": 1
                                    },
                                    "_score": 0.29618382,
                                    "_source": {
                                        "id": "J15.0",
                                        "description": "test one world"
                                    }
                                },
                                {
                                    "_index": "myindexedge64170045",
                                    "_type": "_doc",
                                    "_id": "1",
                                    "_nested": {
                                        "field": "codes",
                                        "offset": 4
                                    },
                                    "_score": 0.29618382,
                                    "_source": {
                                        "id": "J15.3",
                                        "description": "hello world J18 "
                                    }
                                },
                                {
                                    "_index": "myindexedge64170045",
                                    "_type": "_doc",
                                    "_id": "1",
                                    "_nested": {
                                        "field": "codes",
                                        "offset": 5
                                    },
                                    "_score": 0.29618382,
                                    "_source": {
                                        "id": "J15.9",
                                        "description": "hello world new"
                                    }
                                }
                            ]
                        }
                    }
                }
            }

Answer 3

添加另一个答案，因为它是一个不同的问题，第一个答案集中在第一个问题上。

问题是您的第二个查询 test two returns test one world 以及在索引时您使用的是 ngram_analyzer ，它使用的是 标准分析器在 white-spaces 上拆分文本，您的搜索分析器也是 standard，因此如果您在索引文档和搜索词上使用 Analyze API，您会看到它匹配代币：

{
   "text" : "test one world",
   "analyzer" : "standard"
}

并生成令牌

{
    "tokens": [
        {
            "token": "test",
            "start_offset": 0,
            "end_offset": 4,
            "type": "<ALPHANUM>",
            "position": 0
        },
        {
            "token": "one",
            "start_offset": 5,
            "end_offset": 8,
            "type": "<ALPHANUM>",
            "position": 1
        },
        {
            "token": "world",
            "start_offset": 9,
            "end_offset": 14,
            "type": "<ALPHANUM>",
            "position": 2
        }
    ]
}

以及您的搜索字词 test two

{
    "tokens": [
        {
            "token": "test",
            "start_offset": 0,
            "end_offset": 4,
            "type": "<ALPHANUM>",
            "position": 0
        },
        {
            "token": "two",
            "start_offset": 5,
            "end_offset": 8,
            "type": "<ALPHANUM>",
            "position": 1
        }
    ]
}

如您所见，test 令牌出现在您的文档中，因此您得到了该搜索结果。可以通过在查询中使用 AND 运算符来解决，如下所示

搜索查询

{
    "_source": {
        "excludes": [
            "codes"
        ]
    },
    "query": {
        "nested": {
            "path": "codes",
            "query": {
                "bool": {
                    "must": {
                        "multi_match": {
                            "query": "test two",
                            "fields": [
                                "codes.id",
                                "codes.description"
                            ],
                            "operator" :"AND"
                        }
                    }
                }
            },
            "inner_hits": {}
        }
    }
}

和搜索结果

 "hits": [
                                {
                                    "_index": "myindexedge64170045",
                                    "_type": "_doc",
                                    "_id": "1",
                                    "_nested": {
                                        "field": "codes",
                                        "offset": 2
                                    },
                                    "_score": 2.6901608,
                                    "_source": {
                                        "id": "J15.1",
                                        "description": "test two world J15.0"
                                    }
                                },
                                {
                                    "_index": "myindexedge64170045",
                                    "_type": "_doc",
                                    "_id": "1",
                                    "_nested": {
                                        "field": "codes",
                                        "offset": 3
                                    },
                                    "_score": 2.561376,
                                    "_source": {
                                        "id": "J15.2",
                                        "description": "test two three world J15"
                                    }
                                }
                            ]
                        }
                    }
                }
            }

弹性搜索边缘 ngram 未返回所有预期结果

Elastic search edge ngram not returning all expected results

autocomplete

elasticsearch

elasticsearch-mapping

elasticsearch-query

elasticsearch-analyzers