Logstash 到 kibana 多行不适用于自定义消息

Question

我正在尝试通过 logstash 将我的日志文件推送到 elasticsearch 并在 kibana 上显示它。它适用于单行日志记录。但是，当涉及到多行过滤器时，它会失败。
这是我的示例多行日志输入：

2016-06-02T04:02:29,720 INFO  Thread-25-match-entity-bolt a52488cc-316b-402e-af58-3b8a663cd76a STDIO invoke Error processing message:{
  "eid": "f9f16541-4fab-4131-a82e-e3ddf6fcd949",
  "entityInfo": {
    "entityType": "style",
    "defaultLocale": "en-US"
  },
  "systemInfo": {
    "tenantId": "t1"
  },
  "attributesInfo": {
    "externalId": 1514,
    "attributesRead": {
      "IsEntityVariantsValid": false,
      "IsEntityExtensionsValid": false
    },
    "attributesUpdated": {
      "DateAttribute": "2016-06-01T00:00:00.0000000",
      "IsEntitySelfValid": true,
      "IsEntityMetaDataValid": true,
      "IsEntityCommonAttributesValid": true,
      "IsEntityCategoryAttributesValid": true,
      "IsEntityRelationshipsValid": true
    }
  },
  "jsAttributesInfo": {
    "jsRelationship": {
      "entityId": "CottonMaterial001",
      "parentEntityId": "Apparel",
      "category": "Apparel",
      "categoryName": "Apparel",
      "categoryPath": "Apparel",
      "categoryNamePath": "Apparel",
      "variant": "1514",
      "variantPath": "1035/1514",
      "container": "Demo Master",
      "containerName": "Demo Master",
      "containerPath": "DemoOrg/Demo Master/Apparel",
      "organization": "DemoOrg",
      "segment": "A"
    },
    "jsChangeContext": {
      "entityAction": "update",
      "user": "cfadmin",
      "changeAgent": "EntityEditor.aspx",
      "changeAgentType": "PIM",
      "changeInterface": "Entity",
      "sourceTimestamp": "2016-06-01T19:48:19.4162475+05:30",
      "ingestTimestamp": "2016-06-01T19:48:19.4162475+05:30"
    }
  }
}

到目前为止我已经尝试过这些 logstash 配置：

input {
  file {
    path => "path_to_logs/logs.log"
    start_position => "beginning"
  }
}
filter{
    multiline {
      negate => "true"
      pattern => "^%{TIMESTAMP_ISO8601} "
      what => "previous"
    }
    grok{
      match => { "message" => "^%{TIMESTAMP_ISO8601:JigsawTimestamp}%{SPACE}%{LOGLEVEL:JigsawLoglevel}%{SPACE}%{HOSTNAME:ThreadName}%{SPACE}%{UUID:GUID}%{SPACE}%{JAVACLASS:JigsawClassName}%{SPACE}%{WORD:JigsawMethodName}%{SPACE}%{GREEDYDATA:JigsawLogMessage}" } 
    }
}
output {

  if "_grokparsefailure" not in [tags] {
    elasticsearch { 
      hosts => ["localhost:9200"] 
    }
  }
}

第二个：

input {
  file {
    path => "path_to_logs/logs.log"
    start_position => "beginning"
    codec => multiline {
      negate => "true"
      pattern => "^%{TIMESTAMP_ISO8601} "
      what => "previous"
    }
  }
}
filter{
    grok{
      match => { "message" => "^%{TIMESTAMP_ISO8601:JigsawTimestamp}%{SPACE}%{LOGLEVEL:JigsawLoglevel}%{SPACE}%{HOSTNAME:ThreadName}%{SPACE}%{UUID:GUID}%{SPACE}%{JAVACLASS:JigsawClassName}%{SPACE}%{WORD:JigsawMethodName}%{SPACE}%{GREEDYDATA:JigsawLogMessage}" } 
    }
}
output {

  if "_grokparsefailure" not in [tags] {
    elasticsearch { 
      hosts => ["localhost:9200"] 
    }
  }
}

我也试过这个模式：

pattern => "^\s"

但是，none 这有帮助。他们都得到了 _grokparsefailure 标签。我希望 JSON 行成为单个消息的一部分。请指出此过滤器中的错误。

Answer 1

在您的 grok 过滤器中，有几个错误导致您看不到任何日志。

在您的示例数据中 INFO 之后有 2 个空格。
对于字段 JigsawClassName，您正在使用 JAVACLASS 作为输入，这对您的日志来说是错误的。

为什么JAVACLASS错误？

它的实现如下：-

JAVACLASS (?:[a-zA-Z0-9-]+.)+[A-Za-z0-9$]+

根据上述内容，JAVACLASS 要求文本中至少出现一个句点 (.) 符号。但是在您的日志中它只是 STDIO.

用以下内容替换你的 grok 匹配项：-

match => { "message" => "^%{TIMESTAMP_ISO8601:JigsawTimestamp}%{SPACE}%{LOGLEVEL:JigsawLoglevel}%{SPACE}%{SPACE}%{HOSTNAME:ThreadName}%{SPACE}%{UUID:GUID}%{SPACE}%{WORD:JigsawClassName}%{SPACE}%{WORD:JigsawMethodName}%{SPACE}%{GREEDYDATA:JigsawLogMessage}" }

同样为了便于理解，通过添加 stdout 插件将输出重定向到控制台，如下所示：-

output {
if "_grokparsefailure" not in [tags] {
elasticsearch { 
  hosts => ["localhost:9200"] 
}
stdout { codec => rubydebug }
}

让您在使用Logstash处理数据时更容易理解错误。

Logstash 到 kibana 多行不适用于自定义消息

Logstash to kibana multiline not working for custom message

elasticsearch

logstash

logstash-grok

kibana-4