grok 调试器不编译模式

Question

我正在尝试解析 grok 的日志文件。我会将日志粘贴在底部。到目前为止我有这个

\[%{DAY:day} %{MONTH:month} %{MONTHDAY:day} %{TIME:time} %{YEAR:year}\] \[:%{LOGLEVEL:loglevel}\] \[pid %{INT:pid}\] \[client %{IPORHOST:client_ip}:%{INT:client_port}\] %{GREEDYDATA:error_msg}

但它失败了。任何的想法？这是日志文件

[Mon May 16 03:09:08 2016] [error] [client 207.46.13.123] File does not exist: /var/www/html/robots.txt
[Mon May 16 05:55:16 2016] [error] [client 207.46.13.193] File does not exist: /var/www/html/robots.txt
[Mon May 16 14:35:42 2016] [error] [client 207.46.13.123] File does not exist: /var/www/html/robots.txt
[Mon May 16 16:45:15 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico
[Mon May 16 16:45:16 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico
[Mon May 16 17:03:20 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico
[Mon May 16 17:03:20 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico
[Mon May 16 17:03:20 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico
[Mon May 16 17:03:20 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico
[Mon May 16 17:11:39 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico
[Mon May 16 17:11:39 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico
[Mon May 16 17:11:55 2016] [error] [client 212.51.132.183] File does not exist: /var/www/html/CHANGELOG.txt, referer: http://127.0.0.1/CHANGELOG.txt
[Mon May 16 17:12:26 2016] [error] [client 212.51.132.183] File does not exist: /var/www/html/readme.html, referer: http://127.0.0.1/readme.html
[Mon May 16 17:12:58 2016] [error] [client 212.51.132.183] File does not exist: /var/www/html/language, referer: http://127.0.0.1/language/Swedish$%7BIFS%7D&&echo$%7BIFS%7DVxtS%3EDDv2&&tar$%7BIFS%7D/string.js
[Mon May 16 17:22:15 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico, referer: https://my.favsite.com/reports/sc10_trans_history/sc10_trans_history.php
[Mon May 16 17:23:50 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: session_id in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:23:50 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: sessionTableId in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:23:50 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: username in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:24:48 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: session_id in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:24:48 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: sessionTableId in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:24:48 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: username in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:28:38 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico, referer: https://my.favsite.com/reports/sc10_trans_history/sc10_trans_history.php
[Mon May 16 17:30:13 2016] [error] [client 206.174.34.86] PHP Notice:  Array to string conversion in /var/www/html/includes/Customer/Customer.inc on line 173, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:30:13 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: 4f724988-ce32-a743-4dc1-57361c12993c in /var/www/html/Maintenance/MergeAndPurge.php on line 651, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:30:13 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: c695e365-aad2-48b4-b0d4-08e6821d1569 in /var/www/html/Maintenance/MergeAndPurge.php on line 651, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:30:13 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: dc6976bc-8f35-912b-41dc-553d22dc0378 in /var/www/html/Maintenance/MergeAndPurge.php on line 651, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:30:13 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: e1df2d18-aa50-649b-01d3-57367f83b283 in /var/www/html/Maintenance/MergeAndPurge.php on line 651, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:30:13 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: e837a6e4-9a79-a9b1-6b68-5736811c5d7d in /var/www/html/Maintenance/MergeAndPurge.php on line 651, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:30:13 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: fd938bb4-0c67-4013-8363-f5b0a9c01b36 in /var/www/html/Maintenance/MergeAndPurge.php on line 651, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:41:44 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico, referer: https://my.favsite.com/reports/sc10_trans_cust/sc10_trans_cust.php?report_url=sc10_trans_cust.php&default_select_events=&coupon_type=All&print_code=All&seat_codes=&range_select=&range_from=&range_to=&default_sales_source=Internet&default_selected_user=&series_or_events=0&report_tree_id=413&default_start_dt=2016/05/13%2000:00:00&default_end_dt=2016/05/16%2023:59:59&username=JFredrick&sessionid=502umrapqggkocsth42sqo26041&session_id=502umrapqggkocsth42sqo2604&uid=502umrapqggkocsth42sqo2604&where_clause=&exclude_zero_sales=0
[Mon May 16 17:42:58 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: session_id in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:42:58 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: sessionTableId in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:42:58 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: username in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:43:25 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico, referer: https://my.favsite.com/
[Mon May 16 17:43:25 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico, referer: https://my.favsite.com/desktop/desktop.html
[Mon May 16 17:43:26 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico, referer: https://my.favsite.com/
[Mon May 16 17:43:26 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico, referer: https://my.favsite.com/desktop/desktop.html
[Mon May 16 17:44:22 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: session_id in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:44:22 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: sessionTableId in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:44:22 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: username in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 17:44:32 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico, referer: https://my.favsite.com/desktop/desktop.html
[Mon May 16 17:46:27 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico, referer: https://my.favsite.com/desktop/desktop.html
[Mon May 16 18:03:35 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: session_id in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 18:03:35 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: sessionTableId in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 18:03:35 2016] [error] [client 206.174.34.86] PHP Notice:  Undefined index: username in /var/www/html/Application/Authentication.php on line 23, referer: https://my.favsite.com/desktop/desktop.php
[Mon May 16 18:04:58 2016] [error] [client 206.174.34.86] File does not exist: /var/www/html/favicon.ico, referer: https://my.favsite.com/desktop/desktop.html
[Mon May 16 19:05:48 2016] [error] [client 206.174.34.86] PHP Notice:  Array to string conversion in /var/www/html/includes/Customer/Customer.inc on line 173, referer: https://my.favsite.com/desktop/desktop.php

** EIDT **

[%{DAY:day} %{MONTH:month} %{MONTHDAY:day} %{TIME:time} %{YEAR:year}] [%{LOGLEVEL:loglevel}] [client %{IPORHOST :client_ip}] %{GREEDYDATA:error_msg}

给了我更多信息并对其进行了更多解析，但我仍然希望消息和引荐来源网址位于不同的字段中

Answer 1

你可以试试：

filter {
    grok {
        match => {
            "message" =>
            [
                "\[%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}\] \[%{LOGLEVEL:loglevel}\] \[client %{IPORHOST:client_ip}\] %{GREEDYDATA:error_msg}, referer: %{GREEDYDATA:referer}",
                 "\[%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}\] \[%{LOGLEVEL:loglevel}\] \[client %{IPORHOST:client_ip}\] %{GREEDYDATA:error_msg}"
             ]
         }
     }
 }

更新：

这是你的班轮：

\[%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}\] \[%{LOGLEVEL:loglevel}\] \[client %{IPORHOST:client_ip}\] (%{GREEDYDATA:error_msg}, referer: %{GREEDYDATA:referer}|%{GREEDYDATA:error_msg})

grok 调试器不编译模式

grok debugger not compiling the pattern

grok

logstash

logstash-grok

kibana-4

elastic-stack