为存储桶弹性搜索聚合中的每个键设置阈值

Question

我的系统日志包括2个字段，ServiceName和ResponseCode(000 - 999)，response保存了service的处理结果，000表示成功。我需要统计每项服务的成功次数和失败次数，并计算成功率

          "aggs": {
            "group_by_service": {
              "terms": {
                "field": "ServiceName.keyword"
              },
              "aggs": {
                "group_by_count": {
                  "value_count": {
                    "field": "ServiceName.keyword"
                  }
                },
                "group_by_success": {
                  "filter": {
                    "terms": {
                      "ResponseCode": "000"
                    }
                  },
                  "aggs": {
                    "group_by_count_succ": {
                      "value_count": {
                        "field": "ServiceName.keyword"
                      }
                    }
                  }
                },
                "success_percent": {
                  "bucket_script": {
                    "buckets_path": {
                      "numbersucess": "group_by_success>group_by_count_succ",
                      "totalRequests": "group_by_count"
                    },
                    "script": "params.numbersucess / params.totalRequests * 100",
                    "format": "0.00"
                  }
                }
              }
            }
          }

返回的结果：

         "aggregations": {
          "group_by_service": {
            "doc_count_error_upper_bound": 1859,
            "sum_other_doc_count": 94338,
            "buckets": [
              {
                "doc_count": 34361,
                "success_percent": {
                  "value_as_string": "100.00",
                  "value": 100
                },
                "group_by_count": {
                  "value": 34361
                },
                "group_by_success": {
                  "doc_count": 34361,
                  "group_by_count_succ": {
                    "value": 34361
                  }
                },
                "key": "AAA"
              },
              {
                "doc_count": 20474,
                "success_percent": {
                  "value_as_string": "89.27",
                  "value": 89.26931718276839
                },
                "group_by_count": {
                  "value": 20474
                },
                "group_by_success": {
                  "doc_count": 18277,
                  "group_by_count_succ": {
                    "value": 18277
                  }
                },
                "key": "BBB"
              },

我需要为存储桶中的每个键设置阈值：

如果key = AAA，success_percent.value必须大于80
如果key = BBB，success_percent.value必须大于90

...

不符合条件的密钥将从存储桶中删除，这样当我发送提醒邮件时，我只会收到有效的密钥，我该怎么做？

Answer 1

无法访问 bucket_Selector 聚合中的存储桶键。所以我们不能根据条款进行检查。这个问题应该在客户端解决，或者数据应该用预先计算的索引 success_percent

有一种肮脏的方法，即使用多个术语聚合

"aggs": {
    "group_by_service_A": {
      "terms": {
        "field": "ServiceName.keyword",
        "include":"A"  ---> aggregation for "A" only, can also be replaced by filter
      },
      "aggs": {
        "group_by_count": {
          "value_count": {
            "field": "ServiceName.keyword"
          }
        },
        "group_by_success": {
          "filter": {
            "term": {
              "ResponseCode": "000"
            }
          },
          "aggs": {
            "group_by_count_succ": {
              "value_count": {
                "field": "ServiceName.keyword"
              }
            }
          }
        },
        "success_percent": {
          "bucket_script": {
            "buckets_path": {
              "numbersucess": "group_by_success>group_by_count_succ",
              "totalRequests": "group_by_count"
            },
            "script": "params.numbersucess / params.totalRequests * 100",
            "format": "0.00"
          }
        },
        "filter_bucket": {
          "bucket_selector": {
            "buckets_path": {
              "percent":"success_percent"
            },
            "script": "if(params.percent > 20) return true;" --> percent for "A"
          }
        }
      }
    },
    "group_by_service_B": {
      "terms": {
        "field": "ServiceName.keyword",
        "include":"B"---> aggregation for "B" only, can also be replaced by filter
      },
      "aggs": {
        "group_by_count": {
          "value_count": {
            "field": "ServiceName.keyword"
          }
        },
        "group_by_success": {
          "filter": {
            "term": {
              "ResponseCode": "000"
            }
          },
          "aggs": {
            "group_by_count_succ": {
              "value_count": {
                "field": "ServiceName.keyword"
              }
            }
          }
        },
        "success_percent": {
          "bucket_script": {
            "buckets_path": {
              "numbersucess": "group_by_success>group_by_count_succ",
              "totalRequests": "group_by_count"
            },
            "script": "params.numbersucess / params.totalRequests * 100",
            "format": "0.00"
          }
        },
        "filter_bucket": {
          "bucket_selector": {
            "buckets_path": {
              "percent":"success_percent"
            },
            "script": "if(params.percent > 30) return true;" --> percent for "B"
          }
        }
      }
    }
  }

为存储桶弹性搜索聚合中的每个键设置阈值

Set threshold for each key from bucket elasticsearch aggregation

elasticsearch

elasticsearch-aggregation