关于 ELK 中时间戳的问题？

Question

我在 ELK 中遇到时间戳问题，

现在我遇到的问题是 ELK 的 @timestamp 字段在导入旧日志文件时显示当前日期时间。

我需要从日志中的自定义日期时间字段更新@timestamp。

下面是示例日志。

{ "datetime":"2021-08-24 04:13:39,167", "servername":"vm-ws", "serverip":"(null)", "process":"4656", "thread":"4", "level":"DEBUG", "appname":"AcManager", "page":"Program.cs ","method":"ExecuteAsync","line":"63","message":"Starting AcMa Module","otherinfo":{"token":"null","clientip":"null","clientbrowserversion":"null","clienttype":"null"},"moreinfo":"null"}

我在 logstash 中使用了具有以下配置的 grok 过滤器

input {
    stdin {
        type => "stdin-type"
    }
    file {
        type => "json"
        path => [ "/home/testuser/mylogs/*.log", "/home/testuser/mylogs/*/*.log" ]
        start_position => "beginning"
    }
}

filter {
        date {
                match => ["datetime", "yyyy-MM-dd HH:mm:ss"]
                target => ["@timestamp"]
        }

    # Step 1. Extract the JSON String, put it in a temporary field called "payload_raw"
    # Docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

    grok {
        match => {
            "message" => [ "%{JSON:payload_raw}" ]
        }
        pattern_definitions => {
            "JSON" => "{.*$"
        }
    }

    # Step 2. Parse the temporary "payload_raw" field, put the parsed data in a field called "payload"
    # Docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html

    json {
        source => "payload_raw"
        target => "payload"
    }
    
    # Step 3. Remove the temporary "payload_raw" field (and other fields)
    # Docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html

    mutate {
        remove_field => [ "payload_raw","message" ]
    }

#   Tried this but not working
#   date {
#                match => [ "datetime", "yyyy-MM-dd HH:mm:ss" ]
#                target => "@timestamp"
#        }

}

output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        hosts  => "localhost:9200"
    }
}

Answer 1

由于您的“日期时间”字段出现在“有效负载”字段中，您需要以这种方式提及该字段：

date {
            match => [ "[payload][datetime]", "yyyy-MM-dd HH:mm:ss,SSS" ]
            target => "@timestamp"
    }

这应该是正确的脚本：

    input {
    stdin {
        type => "stdin-type"
    }
    file {
        type => "json"
        path => [ "/home/testuser/mylogs/*.log", "/home/testuser/mylogs/*/*.log" ]
        start_position => "beginning"
    }
}

filter {

    # Step 1. Extract the JSON String, put it in a temporary field called "payload_raw"
    # Docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

    grok {
        match => {
            "message" => [ "%{JSON:payload_raw}" ]
        }
        pattern_definitions => {
            "JSON" => "{.*$"
        }
    }

    # Step 2. Parse the temporary "payload_raw" field, put the parsed data in a field called "payload"
    # Docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html

    json {
        source => "payload_raw"
        target => "payload"
    }
    
    # Step 3. Remove the temporary "payload_raw" field (and other fields)
    # Docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html

    mutate {
        remove_field => [ "payload_raw","message" ]
    }

#   Try this
   date {
                match => [ "[payload][datetime]", "yyyy-MM-dd HH:mm:ss,SSS" ]
                target => "@timestamp"
        }


}

output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        hosts  => "localhost:9200"
    }
}

关于 ELK 中时间戳的问题？

Issue regarding Timestamp in ELK?

logging

grok

logstash

kibana

elk