在日志模式中提供多个容器名称以从中抓取数据
Give multiple container names in logs pattern to scrape data from
我已经在 K8s 集群中设置了 EFK 堆栈。目前 fluentd 正在 从所有容器中抓取 日志。
我希望它只从容器 A
、B
、C
和 D
.
中抓取日志
如果我有一些带有 as A-app
的前缀,我可以做如下的事情。
"fluentd-inputs.conf": "# HTTP input for the liveness and readiness probes
<source>
@type http
port 9880
</source>
# Get the logs from the containers running in the node
<source>
@type tail
path /var/log/containers/*-app.log // what can I put here for multiple different containers
# exclude Fluentd logs
exclude_path /var/log/containers/*fluentd*.log
pos_file /opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos
tag kubernetes.*
read_from_head true
<parse>
@type json
</parse>
</source>
# enrich with kubernetes metadata
<filter kubernetes.**>
@type kubernetes_metadata
</filter>
要仅从特定 Pods 抓取日志,您可以使用:
path /var/log/containers/POD_NAME_1*.log,/var/log/containers/POD_NAME_2*.log,.....,/var/log/containers/POD_NAME_N*.log
要从特定 Pods 中的特定容器中抓取日志,您可以使用:
path /var/log/containers/POD_NAME_1*CONTAINER_NAME*.log,/var/log/containers/POD_NAME_2*CONTAINER_NAME*.log,.....,/var/log/containers/POD_NAME_N*CONTAINER_NAME*.log
我创建了一个简单示例来说明其工作原理。
要从 app-1
Pod 中的 web-1
容器中抓取日志,并从 app-2
Pod 中的所有容器中抓取日志,您可以使用:
path /var/log/containers/app-1*web-1*.log,/var/log/containers/app-2*.log
$ kubectl logs -f fluentd-htwn5
...
2021-08-20 13:37:44 +0000 [info]: #0 starting fluentd worker pid=18 ppid=7 worker=0
2021-08-20 13:37:44 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/app-1_default_web-1-ae672aa1405b91701d130da34c54ab3106a8fc4901897ebbf574d03d5ca64eb8.log
2021-08-20 13:37:44 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/app-2-64c99b9f5b-tm6ck_default_nginx-cd1bd7617f04000a8dcfc1ccd01183eafbce9d0155578d8818b27427a4062968.log
2021-08-20 13:37:44 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/app-2-64c99b9f5b-tm6ck_default_frontend-1-e83acc9e7fc21d8e3c8a733e10063f44899f98078233b3238d6b3dc0903db560.log
2021-08-20 13:37:44 +0000 [info]: #0 fluentd worker is now running worker=0
...
我已经在 K8s 集群中设置了 EFK 堆栈。目前 fluentd 正在 从所有容器中抓取 日志。
我希望它只从容器 A
、B
、C
和 D
.
如果我有一些带有 as A-app
的前缀,我可以做如下的事情。
"fluentd-inputs.conf": "# HTTP input for the liveness and readiness probes
<source>
@type http
port 9880
</source>
# Get the logs from the containers running in the node
<source>
@type tail
path /var/log/containers/*-app.log // what can I put here for multiple different containers
# exclude Fluentd logs
exclude_path /var/log/containers/*fluentd*.log
pos_file /opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos
tag kubernetes.*
read_from_head true
<parse>
@type json
</parse>
</source>
# enrich with kubernetes metadata
<filter kubernetes.**>
@type kubernetes_metadata
</filter>
要仅从特定 Pods 抓取日志,您可以使用:
path /var/log/containers/POD_NAME_1*.log,/var/log/containers/POD_NAME_2*.log,.....,/var/log/containers/POD_NAME_N*.log
要从特定 Pods 中的特定容器中抓取日志,您可以使用:
path /var/log/containers/POD_NAME_1*CONTAINER_NAME*.log,/var/log/containers/POD_NAME_2*CONTAINER_NAME*.log,.....,/var/log/containers/POD_NAME_N*CONTAINER_NAME*.log
我创建了一个简单示例来说明其工作原理。
要从 app-1
Pod 中的 web-1
容器中抓取日志,并从 app-2
Pod 中的所有容器中抓取日志,您可以使用:
path /var/log/containers/app-1*web-1*.log,/var/log/containers/app-2*.log
$ kubectl logs -f fluentd-htwn5
...
2021-08-20 13:37:44 +0000 [info]: #0 starting fluentd worker pid=18 ppid=7 worker=0
2021-08-20 13:37:44 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/app-1_default_web-1-ae672aa1405b91701d130da34c54ab3106a8fc4901897ebbf574d03d5ca64eb8.log
2021-08-20 13:37:44 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/app-2-64c99b9f5b-tm6ck_default_nginx-cd1bd7617f04000a8dcfc1ccd01183eafbce9d0155578d8818b27427a4062968.log
2021-08-20 13:37:44 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/app-2-64c99b9f5b-tm6ck_default_frontend-1-e83acc9e7fc21d8e3c8a733e10063f44899f98078233b3238d6b3dc0903db560.log
2021-08-20 13:37:44 +0000 [info]: #0 fluentd worker is now running worker=0
...