匹配 Filebeat 多行模式中完整单词的模式

Question

我在 filebeat.yml 中使用 Filebeat 多行模式，它从单个文件中获取输入，如下所示：

2016-10-06 14:36:00.419 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : requestStartIdentifier: Identifier
2016-10-06 14:36:00.419 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : requestUri: 
2016-10-06 14:36:00.420 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : HttpServletRequest:
 ContentType: text/xml; charset=utf-8
 ContextPath: 
 LocalAddr: 
 LocalName: 
 PathInfo: 
 PathTranslated: 
 QueryString: 
 RequestURI: 
 RequestURL: 
 RemoteHost: 
 ServletPath: 
 Header: Host: 
 Header: Content-Length: 
 Header: Accept-Encoding: 
 Header: SOAPAction: ""
 Header: User-Agent: Apache-HttpClient/4.2.1 
 Header: Content-Type: text/xml; charset=utf-8
 Header: Connection: Keep-Alive
 Header: Accept: text/xml
 
2016-10-06 14:36:00.420 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : uri: , request:
<env:Envelope></env:Envelope>

2016-10-06 14:36:00.419 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : requestStartIdentifier: Identifier
2016-10-06 14:36:00.419 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : requestUri: 
2016-10-06 14:36:00.420 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : HttpServletRequest:
 ContentType: text/xml; charset=utf-8
 ContextPath: 
 LocalAddr: 
 LocalName: 
 PathInfo: 
 PathTranslated: 
 QueryString: 
 RequestURI: 
 RequestURL: 
 RemoteHost: 
 ServletPath: 
 Header: Host: 
 Header: Content-Length: 
 Header: Accept-Encoding: 
 Header: SOAPAction: ""
 Header: User-Agent: Apache-HttpClient/4.2.1 
 Header: Content-Type: text/xml; charset=utf-8
 Header: Connection: Keep-Alive
 Header: Accept: text/xml
 
2016-10-06 14:36:00.420 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : uri: , request:
<env:Envelope></env:Envelope>

filebeat.yml

multiline:
pattern: Identifier
negate: true
match: after

我使用上面的配置来匹配行中的'Identifier'。输出应符合要求

event -1 :
2016-10-06 14:36:00.419 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : requestStartIdentifier: Identifier
2016-10-06 14:36:00.419 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : requestUri: 
2016-10-06 14:36:00.420 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : HttpServletRequest:
 ContentType: text/xml; charset=utf-8
 ContextPath: 
 LocalAddr: 
 LocalName: 
 PathInfo: 
 PathTranslated: 
 QueryString: 
 RequestURI: 
 RequestURL: 
 RemoteHost: 
 ServletPath: 
 Header: Host: 
 Header: Content-Length: 
 Header: Accept-Encoding: 
 Header: SOAPAction: ""
 Header: User-Agent: Apache-HttpClient/4.2.1 
 Header: Content-Type: text/xml; charset=utf-8
 Header: Connection: Keep-Alive
 Header: Accept: text/xml
 
2016-10-06 14:36:00.420 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : uri: , request:
<env:Envelope></env:Envelope>

event -2 :
2016-10-06 14:36:00.419 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : requestStartIdentifier: Identifier
2016-10-06 14:36:00.419 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : requestUri: 
2016-10-06 14:36:00.420 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : HttpServletRequest:
 ContentType: text/xml; charset=utf-8
 ContextPath: 
 LocalAddr: 
 LocalName: 
 PathInfo: 
 PathTranslated: 
 QueryString: 
 RequestURI: 
 RequestURL: 
 RemoteHost: 
 ServletPath: 
 Header: Host: 
 Header: Content-Length: 
 Header: Accept-Encoding: 
 Header: SOAPAction: ""
 Header: User-Agent: Apache-HttpClient/4.2.1 
 Header: Content-Type: text/xml; charset=utf-8
 Header: Connection: Keep-Alive
 Header: Accept: text/xml
 
2016-10-06 14:36:00.420 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : uri: , request:
<env:Envelope></env:Envelope>

Answer 1

根据您的示例输入，我们似乎可以使用包含 requestStartIdentifier: Identifier 的行来表示新事件的开始。我使用 https://play.golang.org/p/BZ2ujeOZZ- 来测试不同的多行参数。

Filebeat 配置：

filebeat:
  prospectors:
    - input_type: log
      paths:
        - input.txt
      multiline:
        pattern: 'requestStartIdentifier: Identifier$'
        negate:  true
        match:   after

output:
  console:
    pretty: true

Filebeat 输出（扩展了换行符）：

{
  "@timestamp": "2016-10-06T21:51:27.244Z",
  "beat": {
    "hostname": "host",
    "name": "host"
  },
  "input_type": "log",
  "message": "2016-10-06 14:36:00.419 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : requestStartIdentifier: Identifier
2016-10-06 14:36:00.419 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : requestUri: 
2016-10-06 14:36:00.420 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : HttpServletRequest:
    ContentType: text/xml; charset=utf-8
    ContextPath: 
    LocalAddr: 
    LocalName: 
    PathInfo: 
    PathTranslated: 
    QueryString: 
    RequestURI: 
    RequestURL: 
    RemoteHost: 
    ServletPath: 
    Header: Host: 
    Header: Content-Length: 
    Header: Accept-Encoding: 
    Header: SOAPAction: \"\"
    Header: User-Agent: Apache-HttpClient/4.2.1 
    Header: Content-Type: text/xml; charset=utf-8
    Header: Connection: Keep-Alive
    Header: Accept: text/xml

2016-10-06 14:36:00.420 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : uri: , request:
\u003cenv:Envelope\u003e\u003c/env:Envelope\u003e
",
  "offset": 962,
  "source": "input.txt",
  "type": "log"
}
{
  "@timestamp": "2016-10-06T21:51:27.244Z",
  "beat": {
    "hostname": "host",
    "name": "host"
  },
  "input_type": "log",
  "message": "2016-10-06 14:36:00.419 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : requestStartIdentifier: Identifier
2016-10-06 14:36:00.419 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : requestUri: 
2016-10-06 14:36:00.420 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : HttpServletRequest:
    ContentType: text/xml; charset=utf-8
    ContextPath: 
    LocalAddr: 
    LocalName: 
    PathInfo: 
    PathTranslated: 
    QueryString: 
    RequestURI: 
    RequestURL: 
    RemoteHost: 
    ServletPath: 
    Header: Host: 
    Header: Content-Length: 
    Header: Accept-Encoding: 
    Header: SOAPAction: \"\"
    Header: User-Agent: Apache-HttpClient/4.2.1 
    Header: Content-Type: text/xml; charset=utf-8
    Header: Connection: Keep-Alive
    Header: Accept: text/xml

2016-10-06 14:36:00.420 DEBUG 29695 --- [XNIO-2 task-2] c.a.a.s.endpoint.endone.server  : uri: , request:
\u003cenv:Envelope\u003e\u003c/env:Envelope\u003e",
  "offset": 1923,
  "source": "input.txt",
  "type": "log"
}

匹配 Filebeat 多行模式中完整单词的模式

pattern to match complete word in Filebeat multi line pattern

filebeat