如何将缺失的元数据添加到 Google 日志代理

Question

使用基于 this I've created a pod on GKE. The agent 的 docker 图像将通过 TCP 监听流畅的事件（我的其他应用程序 pods 将发送事件），然后将这些日志转发到 Google 云日志记录。因为这些事件缺少一些元数据。我该如何添加这些缺失的信息？

(symfony 应用)--[monolog]-->(google-fluentd-agent)-->(云日志)

google-fluentd.conf:

<match fluent.**>
  type null
</match>

# TCP Connections for fluentd aware applications.
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match **>
  type google_cloud
  # Set the chunk limit conservatively to avoid exceeding the GCL limit
  # of 10MiB per write request.
  buffer_chunk_limit 2M
  # Cap the combined memory usage of this buffer and the one below to
  # 2MiB/chunk * (24 + 8) chunks = 64 MiB
  buffer_queue_limit 24
  # Never wait more than 5 seconds before flushing logs in the non-error case.
  flush_interval 5s
  # Never wait longer than 30 seconds between retries.
  max_retry_wait 30
  # Disable the limit on the number of retries (retry forever).
  disable_retry_limit
</match>

Google 中的事件记录缺少数据：

{
  metadata: {
    projectId: "my-project"      
    serviceName: "container.googleapis.com"      
    zone: "us-central1-a"      
    labels: {
      container.googleapis.com/cluster_name: "app-staging-a"        
      compute.googleapis.com/resource_type: "instance"        
      compute.googleapis.com/resource_name: "cluster-fluentd-1dom0"        
      container.googleapis.com/instance_id: "296757089355968949"        
      container.googleapis.com/pod_name: ""        
      compute.googleapis.com/resource_id: "296757089355968949"        
      container.googleapis.com/namespace_name: ""        
      container.googleapis.com/container_name: ""        
    }
    timestamp: "2016-05-16T00:25:37.000Z"      
    projectNumber: "10568438715"      
  }
  insertId: "94dadf6548d"    
  log: "symfony.php"    
  structPayload: {
    context: {
      stack: [33]       
      file: "classes.php"        
      type: 16384        
      line: 4156        
      level: 28928        
    }
    level: "INFO"      
    message: "Using an instance of "This_Function_Method" for function "some_stuff" is deprecated."      
  }
}

Answer 1

google_cloud 输出插件 attempts to parse the namespace, pod, and container names out of the name of the log stream coming into it。在正常设置中，这是可行的，因为磁盘上来自每个容器的 stdout/stderr 的文件都是这样命名的。为了获得类似的解析行为，您必须类似地制作发送到 fluentd 的日志流名称（或在自定义 fluentd 插件中实现您自己的逻辑）。

如何将缺失的元数据添加到 Google 日志代理

How to add missing metadata to Google logging agent

symfony

google-cloud-platform

google-kubernetes-engine

stackdriver