Fluentbit get Docker Kubernetes 中的日志(Systemd)无法正常工作
Fluentbit get Docker Logs(Systemd) in Kubernetes not working
我正在尝试在 Kubernetes 中配置 Fluentbit 以从应用程序 PODs/Docker 容器获取日志并使用 GELF 格式将此日志消息发送到 Graylog,但这不起作用。
在下面查看我的堆栈:
输入
- Docker 版本 1.13.1
- Docker 日志格式=> JSON
- Docker 日志驱动程序 => Journald => systemd
- Fluent-bit 1.3 运行 作为 Kubernetes 中的 Daemonset
- Kubernetes 1.17
- OS主持人:分OS7
输出
- 消息输出格式:GELF 1.1
- 集中式日志 => Graylog 3
问题是 fluentbit 没有从 systemd 读取日志我在两个输出(Systemd,Stdout)中都没有得到任何日志,STDOUT 只是为了帮助排除故障。
我不知道为什么我无法从 systemd 读取。
我完全按照文档
https://docs.fluentbit.io/manual/input/systemd
我的 K8S 配置:
流利位-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: log
labels:
k8s-app: fluent-bit
data:
# Configuration files: server, input, filters and output
# ======================================================
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level debug
Daemon off
@INCLUDE input-systemd.conf
@INCLUDE output-stdout.conf
input-systemd.conf: |
[INPUT]
Name systemd
Tag host.*
Parser json
Systemd_Filter _SYSTEMD_UNIT=docker.service
output-graylog.conf: |
[OUTPUT]
Name gelf
Match *
Host 10.142.15.214
Port 12201
Mode tcp
Gelf_Short_Message_Key log
output-stdout.conf: |
[OUTPUT]
Name stdout
Match *
流利位-ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: log
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
template:
metadata:
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2020"
prometheus.io/path: /api/v1/metrics/prometheus
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:1.3.5
imagePullPolicy: Always
ports:
- containerPort: 2020
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
terminationGracePeriodSeconds: 10
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: fluent-bit-config
configMap:
name: fluent-bit-config
serviceAccountName: fluent-bit
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
fluent-bit-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: fluent-bit-read
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluent-bit-read
subjects:
- kind: ServiceAccount
name: fluent-bit
namespace: log
流利位-role.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: fluent-bit-read
rules:
- apiGroups: [""]
resources:
- namespaces
- pods
verbs: ["get", "list", "watch"]
流利位服务-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluent-bit
namespace: log
我的 Fluentbit OUTPUT(STDOUT) 仅用于调试:
$ kubectl logs -f fluent-bit-2bzxb -n log
[2020/02/20 18:54:23] [Warning] [config] I cannot open /fluent-bit/etc/..2020_02_20_18_54_22.252769193/parsers_custom.conf file
[2020/02/20 18:54:23] [ info] [storage] initializing...
[2020/02/20 18:54:23] [ info] [storage] in-memory
[2020/02/20 18:54:23] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/02/20 18:54:23] [ info] [engine] started (pid=1)
[2020/02/20 18:54:23] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443
[2020/02/20 18:54:23] [ info] [filter_kube] local POD info OK
[2020/02/20 18:54:23] [ info] [filter_kube] testing connectivity with API server...
[2020/02/20 18:54:23] [ info] [filter_kube] API server connectivity OK
[2020/02/20 18:54:23] [ info] [sp] stream processor started
问题是使用此配置我没有从 systemd 获取任何日志
您的 Fluent Bit 容器是否可以访问 Systemd 日志路径?
谢谢@edsiper 我修复了我的 Daemonset 添加 "path: /run/log"
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: log
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
template:
metadata:
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2020"
prometheus.io/path: /api/v1/metrics/prometheus
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:1.3.5
imagePullPolicy: Always
ports:
- containerPort: 2020
env:
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: systemdlog
mountPath: /run/log
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
terminationGracePeriodSeconds: 10
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: systemdlog
hostPath:
path: /run/log
- name: fluent-bit-config
configMap:
name: fluent-bit-config
serviceAccountName: fluent-bit
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
没有足够的 Karma 来 post 评论,所以 posting 作为对 @edsiper 的回答:
“您的 Fluent Bit 容器是否可以访问 Systemd 日志路径?”
在默认设置下 - 不 - 它没有。当我试图解决这个问题时,我偶然发现了这个线程:https://github.com/fluent/fluent-bit/issues/497
长话短说:
您需要 运行 fluent-bit 容器作为 root,因为访问日志需要 root 权限
将 docker 中的机器 ID 设置为与根机器中的相同
绑定/run/log/journal:/run/log/journal
所以:
fluent-bit:
image: 'bitnami/fluent-bit:latest'
restart: always
user: root #give root access
network_mode: host
command: /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf
volumes:
- ./service/config/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
- /etc/machine-id:/etc/machine-id:ro #set the machine id
- /run/log/journal:/run/log/journal #give access to logs
然后,在 fluent-bit.conf 中,您需要编辑 INPUT Path
:
[INPUT]
Name systemd
Tag *
Path /run/log/journal
Systemd_Filter _SYSTEMD_UNIT=docker.service
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
我正在尝试在 Kubernetes 中配置 Fluentbit 以从应用程序 PODs/Docker 容器获取日志并使用 GELF 格式将此日志消息发送到 Graylog,但这不起作用。
在下面查看我的堆栈:
输入
- Docker 版本 1.13.1
- Docker 日志格式=> JSON
- Docker 日志驱动程序 => Journald => systemd
- Fluent-bit 1.3 运行 作为 Kubernetes 中的 Daemonset
- Kubernetes 1.17
- OS主持人:分OS7
输出
- 消息输出格式:GELF 1.1
- 集中式日志 => Graylog 3
问题是 fluentbit 没有从 systemd 读取日志我在两个输出(Systemd,Stdout)中都没有得到任何日志,STDOUT 只是为了帮助排除故障。
我不知道为什么我无法从 systemd 读取。 我完全按照文档 https://docs.fluentbit.io/manual/input/systemd
我的 K8S 配置:
流利位-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: log
labels:
k8s-app: fluent-bit
data:
# Configuration files: server, input, filters and output
# ======================================================
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level debug
Daemon off
@INCLUDE input-systemd.conf
@INCLUDE output-stdout.conf
input-systemd.conf: |
[INPUT]
Name systemd
Tag host.*
Parser json
Systemd_Filter _SYSTEMD_UNIT=docker.service
output-graylog.conf: |
[OUTPUT]
Name gelf
Match *
Host 10.142.15.214
Port 12201
Mode tcp
Gelf_Short_Message_Key log
output-stdout.conf: |
[OUTPUT]
Name stdout
Match *
流利位-ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: log
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
template:
metadata:
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2020"
prometheus.io/path: /api/v1/metrics/prometheus
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:1.3.5
imagePullPolicy: Always
ports:
- containerPort: 2020
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
terminationGracePeriodSeconds: 10
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: fluent-bit-config
configMap:
name: fluent-bit-config
serviceAccountName: fluent-bit
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
fluent-bit-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: fluent-bit-read
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluent-bit-read
subjects:
- kind: ServiceAccount
name: fluent-bit
namespace: log
流利位-role.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: fluent-bit-read
rules:
- apiGroups: [""]
resources:
- namespaces
- pods
verbs: ["get", "list", "watch"]
流利位服务-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluent-bit
namespace: log
我的 Fluentbit OUTPUT(STDOUT) 仅用于调试:
$ kubectl logs -f fluent-bit-2bzxb -n log
[2020/02/20 18:54:23] [Warning] [config] I cannot open /fluent-bit/etc/..2020_02_20_18_54_22.252769193/parsers_custom.conf file
[2020/02/20 18:54:23] [ info] [storage] initializing...
[2020/02/20 18:54:23] [ info] [storage] in-memory
[2020/02/20 18:54:23] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/02/20 18:54:23] [ info] [engine] started (pid=1)
[2020/02/20 18:54:23] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443
[2020/02/20 18:54:23] [ info] [filter_kube] local POD info OK
[2020/02/20 18:54:23] [ info] [filter_kube] testing connectivity with API server...
[2020/02/20 18:54:23] [ info] [filter_kube] API server connectivity OK
[2020/02/20 18:54:23] [ info] [sp] stream processor started
问题是使用此配置我没有从 systemd 获取任何日志
您的 Fluent Bit 容器是否可以访问 Systemd 日志路径?
谢谢@edsiper 我修复了我的 Daemonset 添加 "path: /run/log"
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: log
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
template:
metadata:
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2020"
prometheus.io/path: /api/v1/metrics/prometheus
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:1.3.5
imagePullPolicy: Always
ports:
- containerPort: 2020
env:
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: systemdlog
mountPath: /run/log
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
terminationGracePeriodSeconds: 10
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: systemdlog
hostPath:
path: /run/log
- name: fluent-bit-config
configMap:
name: fluent-bit-config
serviceAccountName: fluent-bit
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
没有足够的 Karma 来 post 评论,所以 posting 作为对 @edsiper 的回答: “您的 Fluent Bit 容器是否可以访问 Systemd 日志路径?” 在默认设置下 - 不 - 它没有。当我试图解决这个问题时,我偶然发现了这个线程:https://github.com/fluent/fluent-bit/issues/497
长话短说:
您需要 运行 fluent-bit 容器作为 root,因为访问日志需要 root 权限
将 docker 中的机器 ID 设置为与根机器中的相同
绑定/run/log/journal:/run/log/journal 所以:
fluent-bit: image: 'bitnami/fluent-bit:latest' restart: always user: root #give root access network_mode: host command: /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf volumes: - ./service/config/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf - /etc/machine-id:/etc/machine-id:ro #set the machine id - /run/log/journal:/run/log/journal #give access to logs
然后,在 fluent-bit.conf 中,您需要编辑 INPUT Path
:
[INPUT]
Name systemd
Tag *
Path /run/log/journal
Systemd_Filter _SYSTEMD_UNIT=docker.service
Systemd_Filter _SYSTEMD_UNIT=kubelet.service