Filebeat 7.10.1 add_docker_metadata 仅添加 container.id

Question

我正在使用安装在主机系统上的 filebeat 7.10.1（不是 docker 容器），运行作为 root 的服务

根据https://www.elastic.co/guide/en/beats/filebeat/current/add-docker-metadata.html 和 https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-container.html

filebeat 配置，filebeat.yml:

filebeat.inputs:
- type: container
  enabled: true
  paths:
    - '/var/lib/docker/containers/*/*.log'
  processors:
    - add_docker_metadata: ~

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

setup.kibana:

output.logstash:
  hosts: ["<logstash_host>:5044"]

启动容器： docker run --rm -d -l my-label --label com.example.foo=bar -p 80:80 nginx

filebeat 获取日志并成功将它们发送到端点（在我的例子中是 logstash，它重新发送到 elasticsearch），但是 filebeat 生成的 json 只包含 container.id 而没有 container.name、container.labels 和 container.image

它看起来像（从 kibana 复制粘贴）：

{
  "_index": "logstash-2021.02.10",
  "_type": "_doc",
  "_id": "s4a4i3cB8j0XLXFVuyMm",
  "_version": 1,
  "_score": null,
  "_source": {
    "@version": "1",
    "ecs": {
      "version": "1.6.0"
    },
    "@timestamp": "2021-02-10T11:33:54.000Z",
    "host": {
      "name": "<some_host>"
    },
    "input": {
      "type": "container"
    },
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "log": {
      .....
    },
    "stream": "stdout",
    "container": {
      "id": "15facae2115ea57c9c99c13df815427669e21053791c7ddd4cd0c8caf1fbdf8c-json.log"
    },
    "agent": {
      "version": "7.10.1",
      "ephemeral_id": "adebf164-0b0d-450f-9a50-11138e519a27",
      "id": "0925282e-319e-49e0-952e-dc06ba2e0c43",
      "name": "<some_host>",
      "type": "filebeat",
      "hostname": "<some_host>"
    }
  },
  "fields": {
    "log.timestamp": [
      "2021-02-10T11:33:54.000Z"
    ],
    "@timestamp": [
      "2021-02-10T11:33:54.000Z"
    ]
  },
  "highlight": {
    "log.logger_name": [
      "@kibana-highlighted-field@gw_nginx@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1612956834000
  ]
}

我做错了什么？如何为发送配置filebeat container.name, container.labels, container.image?

Answer 1

所以在查看 filebeat-debug 和文件系统上的路径之后 - 问题已关闭

原因：符号链接 /var/lib/docker -> /data/docker 产生意外行为

解决方案：

filebeat.inputs:
  - type: container
    enabled: true
    paths:
      - '/data/docker/containers/*/*.log' #use realpath
    processors:
      - add_docker_metadata:
           match_source_index: 3 #subfolder for extract container id from path

Filebeat 7.10.1 add_docker_metadata 仅添加 container.id

Filebeat 7.10.1 add_docker_metadata adds only container.id

docker

filebeat