您如何将来自不同服务的 Fargate EKS Fluent Bit 日志发送到单独的 Cloudwatch 组？

Question

我关注了this guide to configure Fluent Bit and Cloudwatch on my EKS cluster, but currently all of the logs go to one log group. I tried to follow a separate tutorial that used a kubernetes plugin for Fluent Bit to tag the services before the reached the [OUTPUT] configuration. This caused issues because Fargate EKS currently does not handle Fluent Bit [INPUT] configurations as per the bottom of this doc。

有人遇到过这种情况吗？我想将日志拆分成单独的服务。

这是我当前的 YAML 文件。我添加了解析器和过滤器以查看是否可以获得任何其他信息以在 Cloudwatch 上使用。

kind: Namespace
apiVersion: v1
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  parsers.conf: |
    [PARSER]
        Name docker
        Format json
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep On
        
  filters.conf: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
        # Kube_Tag_Prefix kube.var.log.containers.
        Kube_URL https://kubernetes.default.svc:443
        Merge_Log On
        Merge_Log_Key log_processed
        Use_Kubelet true
        Buffer_Size 0
        Dummy_Meta true
  
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   *
        region us-east-1
        log_group_name fluent-bit-cloudwatch2
        log_stream_prefix from-fluent-bit-
        auto_create_group On

Answer 1

所以我发现其实做这个很简单

fluent bit 上输入的默认标签包含您正在登录的服务的名称，因此您实际上可以堆叠多个 [OUTPUT] 块，每个块使用服务名称周围的通配符运算符。这就是我将流发送到不同日志组所要做的全部工作。这是我的 YAML 供参考。

kind: Namespace
apiVersion: v1
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   *logger*
        region us-east-1
        log_group_name logger-fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit-
        auto_create_group On
        
    [OUTPUT]
        Name cloudwatch_logs
        Match   *alb*
        region us-east-1
        log_group_name alb-fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit-
        auto_create_group On

您如何将来自不同服务的 Fargate EKS Fluent Bit 日志发送到单独的 Cloudwatch 组？

How do you send Fargate EKS Fluent Bit Logs from different services to separate Cloudwatch groups?

amazon-cloudwatch

kubernetes

aws-fargate

amazon-eks

fluent-bit