从 URL GROK 模式中提取参数(子字符串)
Extract Parameter (sub-string) from URL GROK Pattern
我有ELK运行用于日志分析。我一切正常。我只想做一些调整。 Whosebug 中的所有 ES/ELK 大神,在此方面提供任何帮助,我将不胜感激。我很乐意请你喝杯咖啡! :D
示例:
URL: /origin-www.domain.com/this/is/a/path?page=2
首先我想得到上面看到的整个路径。
其次,我只想获取参数前的路径:/origina-www.domain.com/this/is/a/path
第三,我只想获取参数:?page=2
第四,我想把日志文件上的时间戳作为kibana上的主要时间戳。目前,kibana 显示的时间戳是处理 ES 的日期和时间。
这是示例条目的样子:
2016-10-19 23:57:32 192.168.0.1 GET /origin-www.example.com/url 200 1144 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-" "-"
这是我的配置:
if [type] == "syslog" {
grok {
match => ["message", "%{IP:client}\s+%{WORD:method}\s+%{URIPATHPARAM:request}\s+%{NUMBER:bytes}\s+%{NUMBER:duration}\s+%{USER-AGENT}\s+%{QS:referrer}\s+%{QS:agent}%{GREEDYDATA}"]
}
date {
match => [ "timestamp", "MMM dd, yyyy HH:mm:ss a" ]
locale => "en"
}
}
ES 版本:5.0.1
日志版本:5.0
基巴纳:5.0
更新:我实际上能够通过使用解决它:
grok {
match => ["message", "%{IP:client}\s+%{WORD:method}\s+%{URIPATHPARAM:request}\s+%{NUMBER:bytes}\s+%{NUMBER:duration}\s+%{USER-AGENT}\s+%{QS:referrer}\s+%{QS:agent}%{GREEDYDATA}"]
}
grok {
match => [ "request", "%{GREEDYDATA:uri_path}\?%{GREEDYDATA:uri_query}" ]
}
kv {
source => "uri_query"
field_split => "&"
target => "query"
}
为了使用日志条目的实际 timestamp
而不是索引时间,您可以使用 date
和 mutate
插件来覆盖现有的 timestamp
值。您可以让 logstash
过滤器看起来像这样:
//filtering your log file
grok {
patterns_dir => ["/pathto/patterns"] <--- you could have a pattern file with such expression LOGTIMESTAMP %{YEAR}%{MONTHNUM}%{MONTHDAY} %{TIME} if you have to change the timestamp format.
match => { "message" => "^%{LOGTIMESTAMP:logtimestamp}%{GREEDYDATA}" }
}
//overriding the existing timestamp with the new field logtimestamp
mutate {
add_field => { "timestamp" => "%{logtimestamp}" }
remove_field => ["logtimestamp"]
}
//inserting the timestamp as UTC
date {
match => [ "timestamp" , "ISO8601" , "yyyyMMdd HH:mm:ss.SSS" ]
target => "timestamp"
locale => "en"
timezone => "UTC"
}
您还可以跟进 Question 了解更多信息。希望对你有帮助。
grok {
match => ["message", "%{IP:client}\s+%{WORD:method}\s+%{URIPATHPARAM:request}\s+%{NUMBER:bytes}\s+%{NUMBER:duration}\s+%{USER-AGENT}\s+%{QS:referrer}\s+%{QS:agent}%{GREEDYDATA}"]
}
grok {
match => [ "request", "%{GREEDYDATA:uri_path}\?%{GREEDYDATA:uri_query}" ]
}
kv {
source => "uri_query"
field_split => "&"
target => "query"
}
我有ELK运行用于日志分析。我一切正常。我只想做一些调整。 Whosebug 中的所有 ES/ELK 大神,在此方面提供任何帮助,我将不胜感激。我很乐意请你喝杯咖啡! :D
示例:
URL: /origin-www.domain.com/this/is/a/path?page=2
首先我想得到上面看到的整个路径。
其次,我只想获取参数前的路径:/origina-www.domain.com/this/is/a/path
第三,我只想获取参数:?page=2
第四,我想把日志文件上的时间戳作为kibana上的主要时间戳。目前,kibana 显示的时间戳是处理 ES 的日期和时间。
这是示例条目的样子:
2016-10-19 23:57:32 192.168.0.1 GET /origin-www.example.com/url 200 1144 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-" "-"
这是我的配置:
if [type] == "syslog" {
grok {
match => ["message", "%{IP:client}\s+%{WORD:method}\s+%{URIPATHPARAM:request}\s+%{NUMBER:bytes}\s+%{NUMBER:duration}\s+%{USER-AGENT}\s+%{QS:referrer}\s+%{QS:agent}%{GREEDYDATA}"]
}
date {
match => [ "timestamp", "MMM dd, yyyy HH:mm:ss a" ]
locale => "en"
}
}
ES 版本:5.0.1 日志版本:5.0 基巴纳:5.0
更新:我实际上能够通过使用解决它:
grok {
match => ["message", "%{IP:client}\s+%{WORD:method}\s+%{URIPATHPARAM:request}\s+%{NUMBER:bytes}\s+%{NUMBER:duration}\s+%{USER-AGENT}\s+%{QS:referrer}\s+%{QS:agent}%{GREEDYDATA}"]
}
grok {
match => [ "request", "%{GREEDYDATA:uri_path}\?%{GREEDYDATA:uri_query}" ]
}
kv {
source => "uri_query"
field_split => "&"
target => "query"
}
为了使用日志条目的实际 timestamp
而不是索引时间,您可以使用 date
和 mutate
插件来覆盖现有的 timestamp
值。您可以让 logstash
过滤器看起来像这样:
//filtering your log file
grok {
patterns_dir => ["/pathto/patterns"] <--- you could have a pattern file with such expression LOGTIMESTAMP %{YEAR}%{MONTHNUM}%{MONTHDAY} %{TIME} if you have to change the timestamp format.
match => { "message" => "^%{LOGTIMESTAMP:logtimestamp}%{GREEDYDATA}" }
}
//overriding the existing timestamp with the new field logtimestamp
mutate {
add_field => { "timestamp" => "%{logtimestamp}" }
remove_field => ["logtimestamp"]
}
//inserting the timestamp as UTC
date {
match => [ "timestamp" , "ISO8601" , "yyyyMMdd HH:mm:ss.SSS" ]
target => "timestamp"
locale => "en"
timezone => "UTC"
}
您还可以跟进 Question 了解更多信息。希望对你有帮助。
grok {
match => ["message", "%{IP:client}\s+%{WORD:method}\s+%{URIPATHPARAM:request}\s+%{NUMBER:bytes}\s+%{NUMBER:duration}\s+%{USER-AGENT}\s+%{QS:referrer}\s+%{QS:agent}%{GREEDYDATA}"]
}
grok {
match => [ "request", "%{GREEDYDATA:uri_path}\?%{GREEDYDATA:uri_query}" ]
}
kv {
source => "uri_query"
field_split => "&"
target => "query"
}