自举索引设置为写入索引，但日志正在写入旧索引

Question

我们运行 kubernetes 上的 Elastic + Fluentbit + Kibana 堆栈用于容器日志，它可以正常工作，基于日期 (new-YYYY-MM-DD) 的每日滚动更新，但在高容量时它导致由于分片大小问题，因此创建了下面提到的 ILM 策略，以便它可以快速滚动。自举索引是可写的，但仍然写入旧索引 (new-YYYY-MM-DD) 而不是新索引 new-YYYY-MM-DD-000001。我已经提到了尝试过的事情，但还没有成功。

Created new policy with following condition:

PUT /_ilm/policy/new_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age" : "10m" #just to test faster, actually want to set it to an hour.
          }
        }
      },
      "delete": {
        "min_age": "20d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Created template:

PUT _template/new_template
{
  "index_patterns": ["new*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "new_policy",
    "index.lifecycle.rollover_alias": "new-alias" 
  }
}

Bootstrapped new index which is creating and rolling over correctly.

PUT /%3Cnew-%7Bnow%2Fd%7D-000001%3E
{
  "aliases": {
    "new-alias":{
      "is_write_index": true 
    }
  }
}

Output plugin section inside fluentbit-configmap

[OUTPUT]
    Name            es
    Match           *
    Host            ${FLUENT_ELASTICSEARCH_HOST}
    Port            ${FLUENT_ELASTICSEARCH_PORT}
    HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
    HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWORD}
    Logstash_Format On
    Logstash_Prefix new
    Trace_Error     On
    Replace_Dots    On
    Retry_Limit     False
    tls             On
    tls.verify      Off

Tried indexing but no luck.

POST _reindex
{
  "source": {
    "index": "new-2021.09.30" 
  },
  "dest": {
    "index": "new-2021.09.30-000001", 
    "op_type": "create" 
  }
}

此外，尝试创建全新的索引模式，但它似乎来自任何系统默认值，而不是将我的 ILM 索引作为新日志的默认值。

alias           index                       filter routing.index routing.search is_write_index
new-alias new-2021.09.30-000001 -      -             -              false
new-alias new-2021.09.30-000002 -      -             -              false
new-alias new-2021.09.30-000003 -      -             -              false
new-alias new-2021.09.30-000004 -      -             -              false
new-alias new-2021.09.30-000005 -      -             -              false
new-alias new-2021.09.30-000006 -      -             -              false
new-alias new-2021.09.30-000007 -      -             -              false
new-alias new-2021.09.30-000008 -      -             -              false
new-alias new-2021.09.30-000009 -      -             -              false
new-alias new-2021.09.30-000010 -      -             -              false
new-alias new-2021.09.30-000011 -      -             -              false
new-alias new-2021.09.30-000012 -      -             -              false
new-alias new-2021.09.30-000013 -      -             -              true

Answer 1

在您的 Fluentbit 配置中，您需要更改以下内容：

Logstash_Format Off
Index new-alias

并删除 Logstash_Prefix new

就是这样。由于 ILM 将负责命名索引并滚动它们，您只需写入 new-alias，ILM 将负责其余的工作。不用再用Fluentbit来补索引名了

自举索引设置为写入索引，但日志正在写入旧索引

Bootstrapped index is set as the write index but logs are getting written to old index

elasticsearch

logstash

kibana

kubernetes

fluent-bit