FluentBit：添加动态 es 索引

Question

我有工作fluent-bit:1.7 我需要输出到 Elasticsearch 并根据 k8s 标签 = name 创建一个动态索引。我想要以下索引约定：

infra-${app_name}-yyyy.mm.dd

示例：infra-mongodb-2021.01.01、infra-postgresql-2021.01.01、infra-kafka-2021.01.01 等...

这是我的过滤器和输出配置：

[FILTER]
    Name             kubernetes
    Match            kube.*
    Merge_Log        Off
    Keep_Log         Off

[OUTPUT]
    Name  es
    Match kube.*
    Host ${ES_HOST}
    Logstash_Format On
    Logstash_Prefix_Key kubernetes['labels']['name']

但它生成以下索引：mongodb-2021.01.01

快完成了，我只需要总是添加infra-前缀。

为了清楚起见，我需要这样的东西：

    Logstash_Prefix_Key infra-${kubernetes['labels']['name']}

Answer 1

我使用 Lua 插件在记录中使用索引名称创建一个字段然后将此字段用作 Logstash_Prefix_Key

Lua 脚本（基于 https://github.com/fluent/fluent-bit/blob/master/scripts/append_tag.lua）：

function append_es_index(tag, timestamp, record)
    new_record = record

    if (record["cluster_name"] ~= nil) then
        es_index = record["cluster_name"]
    else
        es_index = "k8s"
    end

    if (record["kubernetes"] ~= nil) then
        kube = record["kubernetes"]
        if (kube["namespace_name"] ~= nil and string.len(kube["namespace_name"]) > 0) then
            es_index = es_index .. "." .. kube["namespace_name"]
        end
        
        if (kube["labels"] ~= nil) then
            labels = kube["labels"]

            if (labels["app"] ~= nil and string.len(labels["app"]) > 0) then
                es_index = es_index .. "." .. labels["app"]
            elseif (labels["k8s-app"] ~= nil and string.len(labels["k8s-app"]) > 0) then 
                es_index = es_index .. "." .. labels["k8s-app"]
            elseif (labels["name"] ~= nil and string.len(labels["name"]) > 0) then 
                es_index = es_index .. "." .. labels["name"]
            end
        end
    end

    new_record["es_index"] =  es_index

    return 1, timestamp, new_record
end

Fluentbit 过滤器配置：

[FILTER]
    Name                kubernetes
    ...
[FILTER]
    Name record_modifier
    Match *
    Record cluster_name my-test-cluster
[FILTER]
    Name    lua
    Match   *
    script  /fluent-bit/scripts/append_es_index.lua
    call    append_es_index

Answer 2

另一种实现类似结果的方法是为您的 Pods

创建标签或注释

例如

[OUTPUT]
  Name                es
  Logstash_Prefix_Key kubernetes['labels']['log-key']
...

部署yml基于docs

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
        log-key: infra-nginx # log-key label will be used in Logstash_Prefix_Key
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

这样您就可以组合多个“动态”数据，例如命名空间

FluentBit：添加动态 es 索引

FluentBit: add dynamic es index

fluent-bit