如何在 fluentd csv 格式化程序中禁用包含定界符的字段值周围的引号？

Question

我使用 fluentd 从 golang 应用程序收集 CSV 格式的日志。

这是 fluentd 配置文件

<source>
    @type  forward
    @id    app_logs
    @label @mainstream
    port  24224
</source>



<label @mainstream>
   <match **>
      @type file
      @id   v6_logs
    <format>
      @type csv
      fields log_version,day_time,event_id,request_id,app_id,app_version,account_id,country_id,server_name,remote_ip,process_id,thread_id,item_id,message,parameters
      force_quotes false
    </format>
      path         /fluentd/log/app.log
      append       true
  </match>
</label>

我使用 Fluent golang 客户端从应用程序写入日志 https://github.com/fluent/fluent-logger-golang

logger, _ := fluent.New(fluent.Config{FluentPort: 24224, FluentHost: "fluentd"})
defer logger.Close()
tag := "web"
var data = map[string]interface{}{
    "log_version": 6,
    "day_time":    time.Now().UTC().String(),
    "event_id":    1700,
    "request_id":  "54321",
    "account_id":  12345,
    "server_name": hostname,
    "process_id":  os.Getpid(),
    "message":     "Test Message(param1; param2)",
    "parameters":  "value1, value2",
}

error := logger.Post(tag, data)

输出结果是这样的

6,2020-09-23 23:48:44.5731073 +0000 UTC,1700,54321,,,123467,,cabf36399a5c,,1,,,Test Message(param1; param2),"value1,value2"

如何删除“value1,value2”周围的引号（使其作为单独的字段出现）。

Answer 1

我按照此处 https://docs.fluentd.org/v/0.12/developer/plugin-development 中的说明在 ruby 中编写自定义 CSV 格式化程序插件并将文件放在路径 /etc/fluent/plugin/

   require 'fluent/plugin/formatter'
        
   module Fluent::Plugin
   class MyCSVFormatter < Formatter
       # Register MyCSVFormatter as 'my_csv'.
       Fluent::Plugin.register_formatter('my_csv', self)
       config_param :csv_fields, :array, value_type: :string

       # This method does further processing. Configuration parameters can be
       # accessed either via `conf` hash or member variables.
       def configure(conf)
         super
       end

       # This is the method that formats the data output.
      def format(tag, time, record)
         values = []

         # Look up each required field and collect them from the record
         @csv_fields.each do |field|
            if field == "parameters"
                parametervalues = []
                parametervalues = record[field].split(",").map(&:strip)
                values.push(*parametervalues)
                next
            end

            v = record[field]
            values << v.to_s
          end

         # Output by joining the fields with a comma
         values.join(',') + "\n"
      end
    end
   end

更新了 fluentd conf 文件以使用像这样的自定义格式

<label @mainstream>
   <match **>
      @type file
      @id   v6_logs
    <format>
      @type my_csv
      csv_fields log_version,day_time,event_id,request_id,app_id,app_version,account_id,country_id,server_name,remote_ip,process_id,thread_id,item_id,message,parameters
    </format>
      path         /fluentd/log/app.log
      append       true
  </match>
</label>

这会产生所需的输出

6,2020-09-24 06:27:52.1826684 +0000 UTC,1700,54321,,,123467,,hostname,,1,,,Test Message(param1; param2),value1,value2

如何在 fluentd csv 格式化程序中禁用包含定界符的字段值周围的引号？

How to disable quotes around field values that contains delimiters in fluentd csv formatter?

csv

go

export-to-csv

fluentd