无法解析日期字段,尝试了日期格式 [yyyy-MM-dd HH:mm:ss,SSS] 和带有语言环境 [] 的时间戳编号

failed to parse date field, tried both date format [yyyy-MM-dd HH:mm:ss,SSS], and timestamp number with locale []

我正在尝试实施以下答案中描述的解决方案:

但是 Elasticsearch 抛出以下异常,表示它无法解析 @timestamp 字段:

[2015-01-30 12:09:39,513][DEBUG][action.bulk              ] [perfgen 1] [logaggr-2015.01.30][2] failed to execute bulk item (index) index {[logaggr-2015.01.30][logs][c2s5PliTSGKmZSXUWzlkNw], source[{"message":"2015-01-29 17:30:31,579 [ERROR] [pool-1-thread-9] [LogGenerator] invocation count=813,time=2015-01-29 17:30:31,578,metric=-9080142057551045424","@version":"1","@timestamp":"2015-01-30T19:10:53.891Z","host":"perfdev","path":"/home/user/work/elk/logaggr-test/LogAggr_Test.log","logts":"2015-01-29 17:30:31,579","level":"ERROR","thread":"pool-1-thread-9","classname":"LogGenerator","details":"invocation count=813,time=2015-01-29 17:30:31,578,metric=-9080142057551045424"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [@timestamp]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:414)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:648)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:501)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:542)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:491)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:376)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:451)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:157)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:535)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.run(TransportShardReplicationOperationAction.java:434)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [2015-01-30T19:10:53.891Z], tried both date format [yyyy-MM-dd HH:mm:ss,SSS], and timestamp number with locale []
at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:610)
at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:538)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:223)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:404)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "2015-01-30T19:10:53.891Z" is malformed at "T19:10:53.891Z"
at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:604)
... 15 more

如 "message" 中所示,我的日志语句如下所示:

2015-01-29 17:30:31,579 [ERROR] [pool-1-thread-9] [LogGenerator] invocation count=813,time=2015-01-29 17:30:31,578,metric=-9080142057551045424

不确定问题是否与 logstash 配置有关。我的 logstash 过滤器如下所示:

filter {
 grok {
    match => [ "message", "%{TIMESTAMP_ISO8601:logts}%{SPACE}\[%{LOGLEVEL:level}%{SPACE}]%{SPACE}\[%{DATA:thread}]%{SPACE}\[%{DATA:classname}]%{SPACE}%{GREEDYDATA:details}" ]
 }
}

我的 logstash 输出是:

output {
 elasticsearch {
    cluster => "perfgen"
    host => "10.1.1.1"
    port => 9201
    index => "logaggr-%{+YYYY.MM.dd}"
    protocol => "http"
    template => "logaggr-test.json"
    template_name => "logaggr"
 }
}

我的模板 "logaggr-test.json" 是:

{
 "template": "logaggr-*",
 "mappings": {
    "logaggr": {
        "date_detection": false,
        "properties": {
            "_timestamp": { "type": "date", "enabled": true, "store": true },
            "logts": { "type": "date" },
            "level": { "type": "string" },
            "thread": { "type": "string" },
            "classname": { "type": "string" },
            "details": { "type": "string"}
        }
    }
 }
}

我尝试添加默认映射等,但我无法通过解析异常。

为了重申我试图解决的问题,我正在尝试设置 logstash 来解析我的日志文件并将其索引到 Elasticsearch 中。在这个过程中,我想捕获我的日志消息的时间戳,@timestamp(由logstash添加)和_timestamp(由Elasticsearch添加)

感谢任何帮助。

原来我有一个模板是以前的一些测试遗留下来的,它为@timestamp 指定了不同的格式。我删除了模板,现在我可以提取我的日志了。