Grok 过滤器与 bro httplog 数据不匹配

Question

我正在尝试使用 ELK 来可视化 BRO 日志数据。我在网上找到了多个 grok 过滤器，但它总是无法将模式与数据匹配。我尝试使用的过滤器之一是：

grok {
          match => [ "message", "(?<ts>(.*?))\t(?<uid>(.*?))\t(?<id.orig_h>(.*?))\t(?<id.orig_p>(.*?))\t(?<id.resp_h>(.*?))\t(?<id.resp_p>(.*?))\t(?<trans_depth>(.*?))\t(?<method>(.*?))\t(?<bro_host>(.*?))\t(?<uri>(.*?))\t(?<referrer>(.*?))\t(?<user_agent>(.*?))\t(?<request_body_len>(.*?))\t(?<response_body_len>(.*?))\t(?<status_code>(.*?))\t(?<status_msg>(.*?))\t(?<info_code>(.*?))\t(?<info_msg>(.*?))\t(?<filename>(.*?))\t(?<http_tags>(.*?))\t(?<username>(.*?))\t(?<password>(.*?))\t(?<proxied>(.*?))\t(?<orig_fuids>(.*?))\t(?<orig_mime_types>(.*?))\t(?<resp_fuids>(.*?))\t(?<resp_mime_types>(.*))" ]
        }

我尝试获取的兄弟数据如下：

#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   http
#open   2018-11-27-18-31-02
#fields ts  uid id.orig_h   id.orig_p   id.resp_h   id.resp_p   trans_depth method  host    uri referrer    version user_agent  request_body_len    response_body_len   status_code status_msg  info_code   info_msg    tags    username    password    proxied orig_fuids  orig_filenames  orig_mime_types resp_fuids  resp_filenames  resp_mime_types
#types  time    string  addr    port    addr    port    count   string  string  string  string  string  string  count   count   count   string  count   string  set[enum]   string  string  set[string] vector[string]  vector[string]  vector[string]  vector[string]  vector[string]  vector[string]
1543343462.308603   CrJmZi31EU3tUXba3c  10.100.130.72   38396   216.58.217.110  80  1   -   -   -   -   1.1 -   0   219 301 Moved Permanently   -   -   (empty) -   -   -   -   -   -   FXhQ5K1ydVhFnz9Agi  -   text/html
1543344229.051726   CLj9eD4BcFR42BRHV1  10.100.130.72   37452   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   FO0Zko4uvyxeC8LDx4  -   text/plain
1543345395.827176   C6Kdv49oODjjkgeFk   10.100.130.72   37464   169.254.169.254 80  1   -   -   -   -   1.0 -   0   345 404 Not Found   -   -   (empty) -   -   -   -   -   -   FW4NGDCyMNR43J4Hf   -   text/html
1543345691.165771   CNaObqkLN9imdehl4   10.100.130.72   37466   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   FmUSygO8ocHKTN8L3   -   text/plain
1543347316.900516   Ck5CsV2hr56axo3rzl  10.100.130.72   37486   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   FXKDmj3kllpKuJnSkg  -   text/plain
1543348718.870063   CFBClg1jRpmBp4ElYb  10.100.130.72   37506   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   F02j4T12ssIF2tYFF5  -   text/plain
1543348995.827387   CPMwHt2g13sPqdiXE1  10.100.130.72   37508   169.254.169.254 80  1   -   -   -   -   1.0 -   0   345 404 Not Found   -   -   (empty) -   -   -   -   -   -   FsbLPY8A3gpuBkM7l   -   text/html
1543350095.640070   CObHQk2ARejHIWBcgc  10.100.130.72   37518   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   FxCY9C2fOP4dHO2Dkj  -   text/plain

谢谢 -JP

Answer 1

忽略以#开头的行。
我使用了给定的这个 Grok 解析器并且它有效。您可以使用 grokdebugger 来检查解析器。 https://grokdebug.herokuapp.com/
数据源使用空格而不是制表符可能是个问题。验证数据。

我用 => 而不是逗号 (,)。将代码更改为

grok {
  match => [ "message" => "(?<ts>(.*?))\t(?<uid>(.*?))\t(?<id.orig_h>(.*?))\t(?<id.orig_p>(.*?))\t(?<id.resp_h>(.*?))\t(?<id.resp_p>(.*?))\t(?<trans_depth>(.*?))\t(?<method>(.*?))\t(?<bro_host>(.*?))\t(?<uri>(.*?))\t(?<referrer>(.*?))\t(?<user_agent>(.*?))\t(?<request_body_len>(.*?))\t(?<response_body_len>(.*?))\t(?<status_code>(.*?))\t(?<status_msg>(.*?))\t(?<info_code>(.*?))\t(?<info_msg>(.*?))\t(?<filename>(.*?))\t(?<http_tags>(.*?))\t(?<username>(.*?))\t(?<password>(.*?))\t(?<proxied>(.*?))\t(?<orig_fuids>(.*?))\t(?<orig_mime_types>(.*?))\t(?<resp_fuids>(.*?))\t(?<resp_mime_types>(.*))" ]
}

如果错误仍然存在，您也可以使用 csv 解析器而不是 Grok。

Grok 过滤器与 bro httplog 数据不匹配

Grok filter isn't matching to the bro httplog data

logstash

kibana

bro

logstash-grok