如何删除 Logstash Conf 文件中地理坐标中的引号 (" ")

Question

我想在Kibana的地图上表达一条线，地图应该是Geojson结构。我拥有的数据是一组 SQL table 然后我正要像这样使用 Logstash 将它们传输到 Elastic Search

input{ ... }

filter{
    
 if [lat] and [lon] {
        mutate{convert =>  ["lat", "float"]}
        mutate{convert =>  ["lon", "float"]}
        mutate{convert =>  ["for_lat", "float"]}
        mutate{convert =>  ["for_lon", "float"]}

        mutate{
            add_field => {"[location-geotest][type]" => "multilinestring"}
            add_field => {"[location-geotest][coordinates]" => [["%{lon}", "%{lat}"]]}
            add_field => {"[location-geotest][coordinates]" => [["%{for_lon}", "%{for_lat}"]]}
            }
      }
}

但是 logstash conf 文件无法索引 Elasticsearch 上的数据

{
    :status=>400, 
    :action=>["index", {:_id=>"18022", :_index=>"geo_shape_test", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x687994b9>], 
    :response=> {
        "index"=>{
            "_index"=>"geo_shape_test", 
            "_type"=>"_doc", 
            "_id"=>"18022", 
            "status"=>400, 
            "error"=>{
                "type"=>"mapper_parsing_exception", 
                "reason"=>"failed to parse field [location-geotest] of type [geo_shape]", 
                "caused_by"=>{"type"=>"x_content_parse_exception", 
                    "reason"=>"[1:164] [geojson] failed to parse field [coordinates]", 
                    "caused_by"=>{
                        "type"=>"parse_exception", 
                        "reason"=>"geo coordinates must be numbers"
                    }
                }
            }
        }
    }
}

这是 logstash 试图索引的内容之一

{
                 "lat" => 37.567179953757886,
              "gps_id" => 10491,
           "timestamp" => 2020-11-22T06:10:45.000Z,
               "speed" => 17.25745240090587,
                 "lon" => 126.99598717854032,
             "for_lat" => 37.567179953757886,
          "@timestamp" => 2020-11-27T03:54:21.131Z,
             "for_lon" => 126.99598717854032,
            "@version" => "1",
    "location-geotest" => {
        "coordinates" => [
            [0] "[\"126.99598717854032\", \"37.567179953757886\"]",
            [1] "[\"126.99598717854032\", \"37.567179953757886\"]"
        ],
               "type" => "multilinestring"
    }
}

我认为问题是...

 "coordinates" => [
            [0] "[\"126.99598717854032\", \"37.567179953757886\"]",
            [1] "[\"126.99598717854032\", \"37.567179953757886\"]"
        ],

如果我更改部分，它将...

 "coordinates" => [
            [0] [126.99598717854032, 37.567179953757886],
            [1] [126.99598717854032, 37.567179953757886]
        ],

但是我找不到解决方法。

Answer 1

我认为问题如你所说，坐标必须是浮点数而不是字符串。显然，mutate 函数将值转换回字符串。如

所述

https://discuss.elastic.co/t/logstash-mutate-filter-always-stringifies-hash-and-array/25917

他们建议改用 ruby 脚本的解决方案。这已经为线串完成了。

https://discuss.elastic.co/t/geo-shape-geo-link-problems-with-coordinates/179924/4

根据提供的数据，我不明白为什么需要多行字符串？只有两个点应该足以存储为线串。

我用

试过了

filter{
    
 if [lat] and [lon] {

        mutate{
            convert =>  ["lat", "float"]
            convert =>  ["lon", "float"]
            convert =>  ["for_lat", "float"]
            convert =>  ["for_lon", "float"]
            
            add_field => {"[location-geotest][type]" => "linestring"}
        }

      ruby{
          code => "event.set('[location-geotest][coordinates]', [[event.get('lon'), event.get('lat')], [event.get('for_lon'), event.get('for_lat')]])"
      }
  }
}

并得到结果：

    "location-geotest" => {
           "type" => "linestring",
           "coordinates" => [
                 [0] [
                      [0] 126.99598717854032,
                      [1] 37.567179953757886
                     ],
                 [1] [
                      [0] 126.99598717854032,
                      [1] 37.567179953757886
                     ]
           ]
         }

索引正确。

如果你需要多字符串我猜你需要更多的数据并在 ruby 脚本中添加一层数组。

如何删除 Logstash Conf 文件中地理坐标中的引号 (" ")

How To Remove Quotation Marks (" ") In Geo Coordinates On Logstash Conf File

json

geojson

elasticsearch

logstash

kibana