Logstash 解析 csv 日期

Question

我在从 csv 解析日期时遇到问题，我找不到（有人会假设）简单日期的问题 - dd/MM/yy。这是我的 csv 文件的结构：

Date,Key-values,Line Item,Creative,Ad unit,Creative size,Ad server impressions,Ad server clicks,Ad server CTR
04/04/16,prid=DUBAP,Hilton_PostAuth 1,Stop Clicking Around - 300x250,383UKHilton_300x250,300 x 250,31,0,0.00%
04/04/16,prid=DUBAP,Hilton_PostAuth 2,16-0006_Auction_Banners_300x250_cat4,383UKHilton_300x250,300 x 250,59,0,0.00%

和我的 logstash.config 文件：

input {
  file {
    path => "/Users/User/*.csv"
    type => "core2"
    start_position => "beginning"    
  }
}
filter {
  csv {
      columns => ["Date","Key-values","Line Item","Creative","Ad unit","Creative size","Ad server impressions","Ad server clicks","Ad server CTR"]
      separator => ","
  }
  date {
    match => ["Date", "dd/MM/YY"]
  }
  mutate {convert => ["Ad server impressions", "float"]}
  mutate {convert => ["Ad server clicks", "float"]}
  mutate {convert => ["Ad server CTR", "float"]}
}
output {  
    elasticsearch {
        action => "index"
        hosts => "localhost"
        index => "test1"
        workers => 1
    }
    stdout {}
}

我也尝试过将日期组合为 "dd/MM/yy" 但没有成功，日期没有作为日期索引，我只能 select Kibana 中的 @timestamp..

我想一定有一个简单的东西我只是错过了但是至于现在我找不到它..

干杯！

编辑 1：

请在 logstash 启动时查看我的控制台输出以及如何处理数据：

Settings: Default pipeline workers: 4
Pipeline main started
Failed parsing date from field {:field=>"Date", :value=>"Date", :exception=>"Invalid format: \"Date\"", :config_parsers=>"dd/MM/YY", :config_locale=>"default=en_US", :level=>:warn}
2016-05-06T20:32:48.034Z Pawels-MacBook-Air.local Date,Key-values,Line Item,Creative,Ad unit,Creative size,Ad server impressions,Ad server clicks,Ad server CTR
2016-04-03T23:00:00.000Z Pawels-MacBook-Air.local 04/04/16,prid=DUBAP,Hilton_PostAuth 1,Stop Clicking Around - 300x250,383UKHilton_300x250,300 x 250,31,0,0.00%

它仍将其加载到 Elasticsearch 中，但在 Kibana 中没有 'Date' 字段 - 我只能使用 @timestamp

干杯

Answer 1

实际上 date 过滤器的作用是：

The date filter is used for parsing dates from fields, and then using that date or timestamp as the logstash timestamp for the event.

因此，使用该配置，它会读取您的日期并将其用作时间戳字段。如果你想把它作为一个单独的字段使用，配置为：

date {
    match => ["Date", "dd/MM/yy"]
    target => "Date"
}

Logstash 解析 csv 日期

Logstash parsing csv date

csv

datetime

elasticsearch

logstash

kibana