Logstash 日期解析错误（日期字段没有任何时间）

Question

我的数据的日期格式为 yyyy-MM-dd ex : "2015-10-12"

我的 logstash 日期过滤器如下

    input {
            file {
                    path => "/etc/logstash/immport.csv"

                    codec => multiline {
                    pattern => "^S*"
                    negate => true
                    what => "previous"

            }
                    start_position => "beginning"
            }

    }
    filter {
            csv {

                    separator => ","
                    autodetect_column_names => true
                    skip_empty_columns => true
            }

            date {
            match => ["start_date", "yyyy-MM-dd"]
                    target => "start_date"
            }
            mutate {
                 rename => {"start_date" => "[study][startDate]"}
            }

    }
    output {
        elasticsearch {
                action => "index"
                hosts => ["elasticsearch-5-6:9200"]
                index => "immport12"
                document_type => "dataset"
                template => "/etc/logstash/immport-mapping.json"
        template_name => "mapping_template"
        template_overwrite => true
        }

        stdout { codec => rubydebug }
}

但是，我的 es 实例无法解析它，我收到以下错误

"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [study.startDate]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2012-04-17T00:00:00.000Z\" is malformed at \"T00:00:00.000Z\""}}}}}

示例数据行 ][logstash.outputs.elasticsearch] 无法将事件索引到 Elasticsearch。 {:status=>400, :action=>["index", {:_id=>nil, :_index=>"immport_2017_12_02", :_type=>"dataset", :_routing=>nil }, 2017-12-20T08:55:45.367Z 878192e51991 SDY816,HEPSV_COHORT: Participants that received Heplisav,2012-04-17,10.0,Systems Biology Analysis of the response to Licensed Hepatitis B Vaccine (HEPLISAV) ) in specific cell subsets (see companion studies SDY299 and SDY690),Interventional,http://www.immport.org/immport-open/public/study/study/displayStudyDetail/SDY816,,Interventional,Vaccine Response,Homo sapiens,Cell,DNA microarray], :response=>{"index"=>{"_index"=> "immport_2017_12_02"、“_type”=>"dataset"、“_id”=>"AWBzIsBPov62ZQtaldxQ"、"status"=>400、"error"=>{"type" =>"mapper_parsing_exception", "reason"=>"failed to parse [study.startDate]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2012-04-17T00:00:00.000Z\" 在 \"T00:00:00.000Z\""}}}}}

处格式错误

我希望我的 logstash 以这种格式输出日期 yyyy-MM-dd 没有时间戳映射模板

"startDate": {
         "type": "date",
        "format": "yyyy-MM-dd"
 },

Answer 1

我参考您的 logstash conf 文件在我的机器上试过了，它运行良好。

My Logstash conf file :

input {
    file {
        path => "D:\testdata\stack.csv"
        codec => multiline {
            pattern => "^S*"
            negate => true
            what => "previous"
        }
        start_position => "beginning"
    }
}

filter {
    csv {
        separator => ","
        autodetect_column_names => true
        skip_empty_columns => true
    }
    date {
        match => ["dob", "yyyy-MM-dd"]
        target => "dob"
    }
    mutate {
        rename => {"dob" => "[study][dob]"}
    }
}

output {
    elasticsearch {
        action => "index"
        hosts => ["localhost:9200"]
        index => "stack"
    }
    stdout { codec => rubydebug }
}

CSV file :

id,name,rollno,dob,age,gender,comments
1,hatim,88,1992-07-30,25,male,qsdsdadasd asdas das dasd asd asd asd as dd sa d
2,hatim,89,1992-07-30,25,male,qsdsdadasd asdas das dasd asd asd asd as dd sa d

Elasticsearch document after indexing :

{
    "_index": "stack",
    "_type": "doc",
    "_id": "wuBTeGABQ7gwBQSQTX1q",
    "_score": 1,
    "_source": {
        "path": """D:\testdata\stack.csv""",
        "study": {
            "dob": "1992-07-29T18:30:00.000Z"
        },
        "@timestamp": "2017-12-21T09:06:52.465Z",
        "comments": "qsdsdadasd asdas das dasd asd asd asd as dd sa d",
        "gender": "male",
        "@version": "1",
        "host": "INMUCHPC03284",
        "name": "hatim",
        "rollno": "88",
        "id": "1",
        "message": "1,hatim,88,1992-07-30,25,male,qsdsdadasd asdas das dasd asd asd asd as dd sa d\r",
        "age": "25"
    }
}

一切都很完美。看看这个例子是否可以帮助你。

Answer 2

问题是我将 logstash 映射模板名称更改为新名称，但我没有删除旧模板文件，因此索引仍指向旧模板文件

一旦我删除了旧的模板文件

curl -XDELETE 'http://localhost:9200/_templates/test_template'

它起作用了，所以每当我们使用新模板时，都需要删除旧模板，然后处理记录

Logstash 日期解析错误（日期字段没有任何时间）

Logstash date parsing error ( date field doesn't have any time)

elasticsearch

logstash

logstash-configuration

datefilter