logstash 配置中的变量未被替换

Question

None prefix 中的变量被替换 - 为什么？

它适用于旧版本的 logstash (1.5.4)，但不再适用于 2.3。

logstash.cfg（dumps to s3）中的部分输出过滤器：

output {
  if [bucket] == "bucket1" {
     s3 {
       bucket => "bucket1"
       access_key_id => "****"
       secret_access_key => "****"
       region => "ap-southeast-2"
       prefix => "%{env}/%{year}/%{month}/%{day}/"
       size_file => 50000000 #50mb
       time_file => 1
       codec => json_lines # save log as json line (no newlines)
       temporary_directory => "/var/log/temp-logstash"
       tags => ["bucket1"]
     }
  }
  ..
}

示例数据集（取自标准输出）：

{
    "random_person" => "Kenneth Cumming 2016-04-14 00:53:59.777647",
       "@timestamp" => "2016-04-14T00:53:59.917Z",
             "host" => "192.168.99.1",
             "year" => "2016",
            "month" => "04",
              "day" => "14",
              "env" => "dev",
           "bucket" => "bucket1"
}

以防万一，这里是过滤器：

filter {
  mutate {
    add_field => {
      "request_uri" => "%{[headers][request_path]}"
    }
  }
  grok {
    break_on_match => false # default behaviour is to stop matching after first match, we don't want that
    match => { "@timestamp" => "%{NOTSPACE:date}T%{NOTSPACE:time}Z"} # break timestamp field into date and time
    match => { "date" => "%{INT:year}-%{INT:month}-%{INT:day}"} # break date into year month and day fields
    match => { "request_uri" => "/%{WORD:env}/%{NOTSPACE:bucket}"} # break request uri into environment and bucket fields
  }
  mutate {
      remove_field => ["request_uri", "headers", "@version", "date", "time"]
  }

}

Answer 1

known issue 'prefix' 中不允许字段变量。

logstash 配置中的变量未被替换

Variables in logstash config not substituted

amazon-s3

logstash

logstash-grok

logstash-configuration