Logstash 和弹性搜索：在一个值内拆分值

Question

刚刚开始使用 logstash 和弹性搜索

以下是我的日志：

2015-09-09 16:02:23 GET /NeedA/some1/some2/some3/NeedB/some4/NeedC f=json - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+ Mac+OS+X+10_10_5)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/44.0.2403.157+Safari/537.36 http://localhost:3000/ 200 373 554 46

使用下面的配置文件，我能够分离出 url： /NeedA/some1/some2/some3/NeedB/some4/NeedC

filter {
  grok {
    match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:method} %{URIPATH:url} %{NOTSPACE:querystring} %{NOTSPACE:username} %{IPORHOST:ipaddress} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:scstatus} %{NUMBER:scbytes:int} %{NUMBER:csbytes:int} %{NUMBER:timetaken:int}"]
  }
  date {
    match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
    timezone => "Etc/UCT"
  }
}

问题：如何从 /NeedA/some1/some2/some3/NeedB/some4/NeedC 中分离出 NeedA、NeedB 和 NeedC，并将其作为不同的字段放入弹性搜索

Answer 1

解决方法如下：

grok {
                match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:method} \/%{WORD:fieldA}\/.*\/.*\/.*\/%{WORD:fieldB}\/.*\/%{WORD:fieldC} %{NOTSPACE:querystring} %{NOTSPACE:username} %{IPORHOST:ipaddress} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:scstatus} %{NUMBER:scbytes:int} %{NUMBER:csbytes:int} %{NUMBER:timetaken:int}"]
        }

在你的 grok 中，只需将 %{URIPATH:url} 替换为 \/%{WORD:fieldA}\/.*\/.*\/.*\/%{WORD:fieldB}\/.*\/%{WORD:fieldC}

输出结果：

{
          "message" => "2015-09-09 16:02:23 GET /NeedA/some1/some2/some3/NeedB/some4/NeedC f=json - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_5)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/44.0.2403.157+Safari/537.36 http://localhost:3000/ 200 373 554 46",
         "@version" => "1",
       "@timestamp" => "2015-09-09T16:02:23.000Z",
             "host" => "MyHost.local",
             "path" => "/path/of/test.log",
    "log_timestamp" => "2015-09-09 16:02:23",
           "method" => "GET",
           "fieldA" => "NeedA",
           "fieldB" => "NeedB",
           "fieldC" => "NeedC",
      "querystring" => "f=json",
         "username" => "-",
        "ipaddress" => "127.0.0.1",
        "useragent" => "Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_5)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/44.0.2403.157+Safari/537.36",
          "referer" => "http://localhost:3000/",
         "scstatus" => "200",
          "scbytes" => 373,
          "csbytes" => 554,
        "timetaken" => 46
}

此致，阿兰

Logstash 和弹性搜索：在一个值内拆分值

Logstash and elastic search: Split up values within a value

grok

elasticsearch

logstash