在 logstash 中筛选 json

Question

我有一个 json 文件，里面有这样的记录

{"id":1,"first_name":"Frank","last_name":"Mills","date":"5/31/2014","email":"fmills0@feedburner.com","country":"France","city":"La Rochelle","latitude":"46.1667","longitude":"-1.15"

我正在尝试过滤 logstash 中的字段，但到目前为止没有成功。我尝试了 grok debugger and the grokconstructor 但无法正常工作。我最后一次尝试是

input {
    file{
        path => ["C:/logstash-1.4.2/mock_data.json"]
        type => "json"
        start_position => "beginning"
        sincedb_path => "/dev/null"
  }
}
filter {
  mutate {
    replace => [ "message", "%{message}" ]
  }
  json {
    source => "message"
    remove_field => "message"
  }
  mutate {
    convert => [ "latitude", "float" ]
    convert => [ "longitude","float" ]
  }
  mutate {
     rename => [ "latitude", "[location][lat]", "longitude", "[location][lon]" ]
  }
}

output {
  stdout {
    codec => rubydebug
  } 
  elasticsearch {
    host => "127.0.0.1"
    protocol => "http"
    index => "test35"
  }
}

仅用于纬度和经度，但这不起作用。特别是 Json 上的任何 logstash 教程。对此有任何帮助。特定配置文件的输出是

{
 "message" => "{\"id\":91,\"first_name\":\"Adam\",\"last_name\":\"Carr\",\"date\":\"11/14/2014\",\"email\":\"acarr2i@tinyurl.
com\",\"country\":\"Ghana\",\"city\":\"Mampong\",\"latitude\":\"7.06273\",\"longitude\":\"-1.4001\"},",
      "@version" => "1",
      "@timestamp" => "2015-05-04T19:05:08.409Z",
       "host" => "Toshiba",
       "path" => "C:/logstash-1.4.2/mock_data.json",
        "tags" => [
             [0] "_jsonparsefailure"
    ]
}

已为 Alcanzar 更新

Answer 1

geoip 过滤器用于将 IP 地址的 lat/lon 添加到您的数据中。

把所有的部分放在一起会得到这个：

filter {
  grok {
        match => [ 'message', '(?<body>\"id\":.*\"longitude\":\"[^"]+\")' ]
        add_field => [ "json_body", "{%{body}}" ]
  }
  json {
        source => "json_body"
        remove_field => ["message","body","json_body" ]
  }
  mutate {
    convert => [ "latitude", "float" ]
    convert => [ "longitude","float" ]
  }
  mutate {
     rename => [ "latitude", "[location][lat]", 
       "longitude", "[location][lon]" ]
  }
}

这将生成如下所示的事件：

{
      "@version" => "1",
    "@timestamp" => "2015-05-04T19:48:52.051Z",
          "host" => "xxxxxxxx",
            "id" => 1,
    "first_name" => "Frank",
     "last_name" => "Mills",
          "date" => "5/31/2014",
         "email" => "fmills0@feedburner.com",
       "country" => "France",
          "city" => "La Rochelle",
      "location" => {
        "lat" => 46.1667,
        "lon" => -1.15
    }
}

这应该正是您想要的。

在 logstash 中筛选 json

filter json in logstash

json

logstash

elasticsearch-plugin

kibana