在 Logstash 中分解 json 数组字符串

Question

我有一个 RabbitMQ 以 JSON 格式发送 3 个字段，它正在被 Logstash rabbitmq 输入插件使用。

其中一个字段是 JSON 字符串数组，如下所示：

"content": [
  "1111",
  "2222222",
  "Test 06",
  "3",
  "3232",
  "SomeValue1"
]

如何将该字符串的每个条目都放入一个字段中，以便我可以从可用字段中快速发现并在 Kibana 中可视化？现在我看到 "content" 和完整的字符串。

JSON 数组字符串大小根据另一个字段 eventID 而变化。是否可以根据 eventID 将该字符串中的值动态映射到特定名称？如：

"eventID": 1,
  "content": [
  "name1": "1111",
  "name2": "2222222",
  "name3": "Test 06",
  "name4": "3",
  "name5": "3232",
  "name6": "SomeValue1"
]

"eventID": 2,
  "content": [
  "othername1": "3434",
  "othername2": "Test 10",
  "othername3": "876",
  "othername4": "Some String7"
]

我想在可用字段中输入姓名* 和其他姓名*。任何帮助，将不胜感激。

Answer 1

首先，我暂时假设您的输入已正确解析为数组。出于测试目的，这意味着：

echo '{"eventID":1,"content":["a","b","c","d"]}' | bin/logstash -f test.conf

其中 test.conf 是：

input {
    stdin { codec => json }
}
output {
    stdout { codec => rubydebug }
}

将输出：

{
       "eventID" => 1,
    "@timestamp" => 2017-08-03T19:39:13.054Z,
      "@version" => "1",
          "host" => "xxxxxx.local",
       "content" => [
        [0] "a",
        [1] "b",
        [2] "c",
        [3] "d"
    ]
}

如果是这种情况，那么您需要执行以下操作：

filter {
    if [eventID] == 1 {
        mutate {
            add_field => {
                "eventName" => "type1 event"
                "one0" => "%{[content][0]}"
                "one1" => "%{[content][1]}"
                "one2" => "%{[content][2]}"
                "one3" => "%{[content][3]}"
            }
            remove_field => [ "content" ]
        }
    } else if [eventID] == 2 {
        mutate {
            add_field => {
                "eventName" => "type2 event"
                "two0" => "%{[content][0]}"
                "two1" => "%{[content][1]}"
                "two2" => "%{[content][2]}"
                "two3" => "%{[content][3]}"
            }
            remove_field => [ "content" ]
        }
    }
}

这将生成如下事件：

{
       "eventID" => 1,
    "@timestamp" => 2017-08-03T19:51:02.946Z,
      "@version" => "1",
          "host" => "xxxxxxx.local",
          "one2" => "c",
     "eventName" => "type1 event",
          "one3" => "d",
          "one0" => "a",
          "one1" => "b"
}

在 Logstash 中分解 json 数组字符串

Decompose json array string in Logstash

arrays

json

elasticsearch

logstash

kibana