Logstash 解析器错误，时间戳格式错误

Question

有人能告诉我哪里做错了吗，或者为什么 Logstash 不想解析 ISO8601 时间戳？

我收到的错误信息是

Failed action ... "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2017-03-24 12:14:50\" is malformed at \"17-03-24 12:14:50\""}}

示例日志文件行（IP 地址中的最后一个字节故意替换为 000）

2017-03-24 12:14:50 87.123.123.000 12345678.domain.com GET /smil:stream_17.smil/chunk_ctvideo_ridp0va0r600115_cs211711500_mpd.m4s - HTTP/1.1 200 750584 0.714 "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" https://referrer.domain.com/video/2107 https fra1 "HIT, MISS" 12345678.domain.com

GROK 模式（使用http://grokconstructor.appspot.com/do/match 验证）

RAW %{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{IPV4:clientip}%{SPACE}%{HOSTNAME:http_host}%{SPACE}%{WORD:verb}%{SPACE}\/(.*:)?%{WORD:stream}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:protocol}\/%{NUMBER:httpversion}%{SPACE}%{NUMBER:response}%{SPACE}%{NUMBER:bytes}%{SPACE}%{SECOND:request_time}%{SPACE}%{QUOTEDSTRING:agent}%{SPACE}%{URI:referrer}%{SPACE}%{WORD}%{SPACE}%{WORD:location}%{SPACE}%{QUOTEDSTRING:cache_status}%{SPACE}%{WORD:account}%{GREEDYDATA}

Logstash 配置（输入端）：

input {
    file {
      path => "/subfolder/logs/*"
      type => "access_logs"
      start_position => "beginning"
    }
}
filter {
    # skip first two lines in log file with comments
    if [message] =~ /^#/ {
        drop { }
    }

    grok {
        patterns_dir => ["/opt/logstash/patterns"]
        match => { "message" => "%{RAW}" }
    }

    date {
        match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]
        locale => "en"
    }

    # ... (rest of the config omitted for readability)
}

Answer 1

所以我很确定这是由字段 timestamp 映射到 Elasticsearch 中它不解析的类型引起的。如果您 post 您的索引映射，我很乐意查看。

注意：您可以通过添加 remove_field 快速解决此问题，因为如果 date 过滤器成功，该字段的值将被拉入 @timestamp。现在您在两个字段中存储了相同的值。这样您就不必担心该字段的映射。 :)

date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]
    locale => "en"
    remove_field => [ "timestamp" ]
}

Logstash 解析器错误，时间戳格式错误

Logstash parser error, timestamp is malformed

logstash

logstash-grok

logstash-file

logstash-configuration