filebeat 不能在一个实例中使用多个 input/output 配置文件吗?
Can't filebeat use multiple input/output config files in one instance?
想要同时部署具有 3 个日志定义的 filebeat。发送到不同的输出目标。
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
labels:
k8s-app: filebeat
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.10.0
args: [
"-c", "/etc/logs1.yml",
"-c", "/etc/logs2.yml",
"-c", "/etc/logs3.yml",
"-e",
]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config-logs1
mountPath: /etc/logs1.yml
subPath: filebeat-logs1.yml
readOnly: true
- name: config-logs2
mountPath: /etc/logs2.yml
subPath: logs2.yml
readOnly: true
- name: config-logs3
mountPath: /etc/logs3.yml
subPath: logs3.yml
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true
volumes:
- name: config-logs1
configMap:
defaultMode: 0600
name: configmap-logs1
- name: config-logs2
configMap:
defaultMode: 0600
name: configmap-logs2
- name: config-logs3
configMap:
defaultMode: 0600
name: configmap-logs3
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
logs1 的配置图
data:
filebeat-logs1.yml: |-
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/logs1.json
output.logstash:
hosts: ["logstash-logs1.default.svc.cluster.local:5044"]
logs2 的 configmap
data:
filebeat-logs2.yml: |-
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/logs2.json
output.logstash:
hosts: ["logstash-logs2.default.svc.cluster.local:5044"]
logs3 的配置图
data:
filebeat-logs3.yml: |-
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/logs3.json
output.logstash:
hosts: ["logstash-logs3.default.svc.cluster.local:5044"]
当每个日志文件改变时,每次只发送到第三个logs3的输出logstash-logs3.default.svc.cluster.local:5044
。但是可以得到三个logs1.json/logs2.json/logs3.json个文件的数据。
在这种情况下,filebeat 不能在一台机器上使用多个输出吗?
您可以有任意多的输入,但您可以 only have one output,您需要将您的日志发送到一个 logstash,然后您可以从那里将它们发送到其他地方。
Filebeat 不支持同时向多个logstash 服务器发送相同的数据。为此,您必须使用不同的 logstash 服务器配置启动多个 Filebeat 实例。
这是 Filebeat 输出插件的限制
在同一主机启动多个filebeat实例,可以参考this link
将数据发送到 Logstash 服务器和控制台输出同时工作吗?
output.logstash:
hosts: ["localhost:5044"]
output.console:
像这样..
想要同时部署具有 3 个日志定义的 filebeat。发送到不同的输出目标。
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
labels:
k8s-app: filebeat
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.10.0
args: [
"-c", "/etc/logs1.yml",
"-c", "/etc/logs2.yml",
"-c", "/etc/logs3.yml",
"-e",
]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config-logs1
mountPath: /etc/logs1.yml
subPath: filebeat-logs1.yml
readOnly: true
- name: config-logs2
mountPath: /etc/logs2.yml
subPath: logs2.yml
readOnly: true
- name: config-logs3
mountPath: /etc/logs3.yml
subPath: logs3.yml
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true
volumes:
- name: config-logs1
configMap:
defaultMode: 0600
name: configmap-logs1
- name: config-logs2
configMap:
defaultMode: 0600
name: configmap-logs2
- name: config-logs3
configMap:
defaultMode: 0600
name: configmap-logs3
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
logs1 的配置图
data:
filebeat-logs1.yml: |-
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/logs1.json
output.logstash:
hosts: ["logstash-logs1.default.svc.cluster.local:5044"]
logs2 的 configmap
data:
filebeat-logs2.yml: |-
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/logs2.json
output.logstash:
hosts: ["logstash-logs2.default.svc.cluster.local:5044"]
logs3 的配置图
data:
filebeat-logs3.yml: |-
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/logs3.json
output.logstash:
hosts: ["logstash-logs3.default.svc.cluster.local:5044"]
当每个日志文件改变时,每次只发送到第三个logs3的输出logstash-logs3.default.svc.cluster.local:5044
。但是可以得到三个logs1.json/logs2.json/logs3.json个文件的数据。
在这种情况下,filebeat 不能在一台机器上使用多个输出吗?
您可以有任意多的输入,但您可以 only have one output,您需要将您的日志发送到一个 logstash,然后您可以从那里将它们发送到其他地方。
Filebeat 不支持同时向多个logstash 服务器发送相同的数据。为此,您必须使用不同的 logstash 服务器配置启动多个 Filebeat 实例。
这是 Filebeat 输出插件的限制
在同一主机启动多个filebeat实例,可以参考this link
将数据发送到 Logstash 服务器和控制台输出同时工作吗?
output.logstash:
hosts: ["localhost:5044"]
output.console:
像这样..