使用 filebeat 合并 date/time 之间的日志

Question

我正在尝试使用 fileBeat（无 Logstash）

将日志推送到 elasticSearch

我想在单条消息中发送以下日志，但它被分成多条消息，每一行都变成单独的消息

20161014 17:49:09.169 [ERROR] [Thread-2974] some.java.class.:70 - some.java.Exception: write failed. History: [requestHost=123-some.org.com, time=Fri Oct 14 17:49:05 GMT-07:00 2016, exception=java.net.SocketTimeoutException]
[requestHost=123-some.org.com, time=Fri Oct 14 17:49:07 GMT-07:00 2016, exception=java.net.SocketTimeoutException]
[requestHost=123-some.org.com, time=Fri Oct 14 17:49:09 GMT-07:00 2016, exception=java.net.SocketTimeoutException]
 Tried 3 times
        at java.lang.Thread.run(Thread.java:745)
20161014 17:49:09.169 [ERROR] [Thread-3022]

我想合并两个日期之间的所有行（第一行和最后一行）

这是我的 filebeat.yml 片段

 paths:
      - /test.log
      multiline.pattern: '^\[0-9]{8}'
      multiline.negate: true
      multiline.match: after

我需要知道正确的 regex

我试图在不使用 logstash

的情况下解决这个问题

Answer 1

将以下 Filebeat 配置与提供的日志样本结合使用会产生两个事件，其中每条消息都以日期开头。

我运行./filebeat -c filebeat.yml -e -v -d "*"用下面的配置来测试。我还在 Go playground.

上测试了模式

filebeat.yml:

filebeat:
  prospectors:
    - paths: ["input.txt"]
      multiline:
        pattern: '^[0-9]{8}'
        negate:  true
        match:   after
output:
  console:
    pretty: false

输出：

{   
  "@timestamp": "2016-10-17T14:13:31.292Z",
  "beat": {
    "hostname": "host.example.com",
    "name": "host.example.com",
  },  
  "input_type": "log",
  "message": "20161014 17:49:09.169 [ERROR] [Thread-2974] some.java.class.:70 - some.java.Exception: write failed. History: [requestHost=123-some.org.com, time=Fri Oct 14 17:49:05 GMT-07:00 2016, exception=java.net.SocketTimeoutException]\n[requestHost=123-some.org.com, time=Fri Oct 14 17:49:07 GMT-07:00 2016, exception=java.net.SocketTimeoutException]\n[requestHost=123-some.org.com, time=Fri Oct 14 17:49:09 GMT-07:00 2016, exception=java.net.SocketTimeoutException]\n Tried 3 times\n        at java.lang.Thread.run(Thread.java:745)",
  "offset": 519,
  "source": "input.txt",
  "type": "log"
}   
{   
  "@timestamp": "2016-10-17T14:17:21.686Z",
  "beat": {
    "hostname": "host.example.com",
    "name": "host.example.com",
  },  
  "input_type": "log",
  "message": "20161014 17:49:09.169 [ERROR] [Thread-3022]",
  "offset": 563,
  "source": "input.txt",
  "type": "log"
}

使用 filebeat 合并 date/time 之间的日志

Merge logs between date/time using filebeat

elasticsearch

logstash

logstash-grok

filebeat