XML 使用 kibana 和 logstash 在网格中显示数据

Question

我想使用 logstash 和 Kibana 以网格格式显示 XML 数据。使用下面的 conf 文件，我能够将数据显示到网格中，但不能拆分行数据。示例：

输出

logstash.conf 文件：

input {
 file {
path => "C:/ELK Stack/logstash-8.2.0-windows-x86_64/logstash-8.2.0/Test.xml"
start_position => "beginning"  
sincedb_path => "NUL"
codec => multiline {
pattern => "^<?stations.*>"
negate => "true"
what => "previous"
auto_flush_interval => 1
max_lines => 3000   
}}}

filter
{
 xml
   {
    source => "message"
    target => "parsed"
    store_xml => "false"
    xpath => [
        "/stations/station/id/text()", "station_id",
        "/stations/station/name/text()", "station_name"
    ]
}
mutate {
   remove_field => [ "message"]
}
}
output {  
elasticsearch {
    action => "index"
    hosts => "localhost:9200"
     index => "logstash_index123xml"
    workers => 1
}
stdout {
codec => rubydebug
}

}

Answer 1

xpath 总是 return 数组，要关联两个数组的成员，您需要使用 ruby 过滤器。要获得多个事件，您可以使用拆分过滤器来拆分您在 ruby 过滤器中构建的数组。如果您从

开始

<stations> 
<station> 
<id>1</id> 
<name>a</name> 
<id>2</id>
<name>b</name> 
</station> 
</stations>

那么如果你使用

    xml {
        source => "message"
        store_xml => "false"
        xpath => {
            "/stations/station/id/text()" => "[@metadata][station_id]"
            "/stations/station/name/text()" => "[@metadata][station_name]"
        }
        remove_field => [ "message" ]
    }
    ruby {
        code => '
            ids = event.get("[@metadata][station_id]")
            names = event.get("[@metadata][station_name]")
            if ids.is_a? Array and names.is_a? Array y and ids.length == names.length
                a = []
                ids.each_index { |x|
                    a << { "station_name" => names[x], "station_id" => ids[x] }
                }
                event.set("[@metadata][theData]", a)
            end
        '
    }
    if [@metadata][theData] {
        split {
            field => "[@metadata][theData]"
            add_field => {
                "station_name" => "%{[@metadata][theData][station_name]}"
                "station_id" => "%{[@metadata][theData][station_id]}"
            }
        }
    }

您将获得两个活动

{
    "station_name" => "a",
      "station_id" => "1",
    ...
}
{
    "station_name" => "b",
      "station_id" => "2",
    ...
}

XML 使用 kibana 和 logstash 在网格中显示数据

XML data display in grid using kibana and logstash

java

xml

elasticsearch

logstash

kibana