filebeat 不能在一个实例中使用多个 input/output 配置文件吗？

Question

想要同时部署具有 3 个日志定义的 filebeat。发送到不同的输出目标。

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.10.0
          args: [
            "-c", "/etc/logs1.yml",
            "-c", "/etc/logs2.yml",
            "-c", "/etc/logs3.yml",
            "-e",
          ]
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config-logs1
              mountPath: /etc/logs1.yml
              subPath: filebeat-logs1.yml
              readOnly: true
            - name: config-logs2
              mountPath: /etc/logs2.yml
              subPath: logs2.yml
              readOnly: true
            - name: config-logs3
              mountPath: /etc/logs3.yml
              subPath: logs3.yml
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: config-logs1
          configMap:
            defaultMode: 0600
            name: configmap-logs1
        - name: config-logs2
          configMap:
            defaultMode: 0600
            name: configmap-logs2
        - name: config-logs3
          configMap:
            defaultMode: 0600
            name: configmap-logs3
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate

logs1 的配置图

data:
  filebeat-logs1.yml: |-
    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /var/log/logs1.json

    output.logstash:
      hosts: ["logstash-logs1.default.svc.cluster.local:5044"]

logs2 的 configmap

data:
  filebeat-logs2.yml: |-
    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /var/log/logs2.json

    output.logstash:
      hosts: ["logstash-logs2.default.svc.cluster.local:5044"]

logs3 的配置图

data:
  filebeat-logs3.yml: |-
    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /var/log/logs3.json

    output.logstash:
      hosts: ["logstash-logs3.default.svc.cluster.local:5044"]

当每个日志文件改变时，每次只发送到第三个logs3的输出logstash-logs3.default.svc.cluster.local:5044。但是可以得到三个logs1.json/logs2.json/logs3.json个文件的数据。

在这种情况下，filebeat 不能在一台机器上使用多个输出吗？

Answer 1

您可以有任意多的输入，但您可以 only have one output，您需要将您的日志发送到一个 logstash，然后您可以从那里将它们发送到其他地方。

Answer 2

Filebeat 不支持同时向多个logstash 服务器发送相同的数据。为此，您必须使用不同的 logstash 服务器配置启动多个 Filebeat 实例。

这是 Filebeat 输出插件的限制

在同一主机启动多个filebeat实例，可以参考this link

Answer 3

将数据发送到 Logstash 服务器和控制台输出同时工作吗？

output.logstash:
   hosts: ["localhost:5044"]
output.console:

像这样..

filebeat 不能在一个实例中使用多个 input/output 配置文件吗？

Can't filebeat use multiple input/output config files in one instance?

kubernetes

logstash

instance

filebeat

elastic-beats