logstash gork filter error : fetched an invalid config

Question

我正在使用 Filebeat -> Logstash -> Elasticsearch -> Kibana 来概览我的 glassfish 日志文件。

你知道我的 Logstash 过滤器配置有什么问题吗？

我的过滤器配置如下所示：

filter {
  if [type] == "log" {
    grok {
      match => { "message", "(?m)\[\#\|%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:Log Level}\|%{DATA:server_version}\|%{JAVACLASS:Class}\|%{DATA:thread}\|%{DATA:message_detail}\|\#\]" }
      add_field => [ "Log level", "%{LOGLEVEL:Log Level}" ]
    }
  }
syslog_pri { }
date {
  match => {[ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]}
  }
}

当我启动 filebeat 5.0 时

sudo ./filebeat -e -c filebeat-logstash.yml -d "publish

ERR Failed to publish events caused by: EOF
2016/11/11 14:53:54.397825 single.go:91: INFO Error publishing events (retrying): EOF
2016/11/11 14:54:09.232146 logp.go:230: INFO Non-zero metrics in the last 30s:     filebeat.harvester.open_files=1 libbeat.logstash.call_count.PublishEvents=5 libbeat.logstash.publish.read_errors=5 libbeat.logstash.publish.write_bytes=5542 libbeat.publisher.published_events=2047 filebeat.harvester.running=1 libbeat.logstash.published_but_not_acked_events=10235 filebeat.harvester.started=1

Logstash 日志

http://pastebin.com/Hh9ECFjd

{:timestamp=>"2016-11-11T14:40:35.247000+0000", :message=>"fetched an invalid config", :config=>"input {\n  lumberjack {\n    port => 5000\n    type => \"log\"\n    ssl => false\n    #ssl_certificate => \"/etc/pki/tls/certs/logstash-forwarder.crt\"\n    #ssl_key => \"/etc/pki/tls/private/logstash-forwarder.key\"\n  }\n}\n\ninput {\n  beats {\n   port => 5044\n   ssl => false\n   # ssl_certificate => \"/etc/pki/tls/certs/logstash-beats.crt\"\n   # ssl_key => \"/etc/pki/tls/private/logstash-beats.key\"\n  }\n}\n\nfilter {\n  if [type] == \"log\" {\n    grok {\n      match => { \"message\", \"(?m)\[\#\|%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:Log Level}\|%{DATA:server_version}\|%{JAVACLASS:Class}\|%{DATA:thread}\|%{DATA:message_detail}\|\#\]\" }\n      add_field => [ \"Log level\", \"%{LOGLEVEL:Log Level}\" ]\n    }\n   }\n    syslog_pri { }\n    date {\n      match => {[ \"timestamp\", \"MMM  d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]}\n   }\n}\n\n\nfilter {\n  if [type] == \"nginx-access\" {\n    grok {\n      match => { \"message\" => \"%{NGINXACCESS}\" }\n    }\n  }\n}\n\noutput {\n  elasticsearch {\n    hosts => [\"http://localhost:9200\"]\n    sniffing => true\n    manage_template => false\n    index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"\n    document_type => \"%{[@metadata][type]}\"\n  }\n}\n\n", :reason=>"Expected one of #, => at line 23, column 27 (byte 461) after filter {\n  if [type] == \"log\" {\n    grok {\n      match => { \"message\"", :level=>:error}

非常感谢。

托马斯

Answer 1

您的配置有几处错误。

grok {
  match => { "message", "(?m)\[\#\|%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:log_level}\|%{DATA:server_version}\|%{JAVACLASS:Class}\|%{DATA:thread}\|%{DATA:message_detail}\|\#\]" }
}

不要在字段名称中使用空格。要解析的字段和 grokstring 需要一个箭头 =>。他们不受支持。您也不需要添加 loglevel 字段，因为 grok 会为您做这件事。

date {
  match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
}

日期匹配中不需要花括号，因为它需要一个数组。

lumberjack input 要求您使用 SSL，您无法使用 ssl => false 禁用它。然而，这适用于节拍输入。

完整配置如下所示：

input {
  lumberjack {
    port => 5000
    type => "log"
    ssl_certificate => "/etc/foo.crt"
    ssl_key => "/etc/foo.key"
  }
}
input {
  beats {
   port => 5044
  ssl => false
    }
}
filter {
if [type] == "log" {
    grok {
      match => { "message" =>  "(?m)\[\#\|%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:log_level}\|%{DATA:server_version}\|%{JAVACLASS:Class}\|%{DATA:thread}\|%{DATA:message_detail}\|\#\]" }
    }
  }
syslog_pri { }
date {
  match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%[@metadata][beat]}-%{+YYYY.MM.dd"
    document_type => "%[@metadata][type]"
  }
}

关于调试配置的另一件事。您发布了您的 logstash 日志，而 logstash 本身会告诉您它无法解释的内容。

:reason=>"Expected one of #, => at line 23, column 27 (byte 461) after filter {
  if [type] == \"log\" {
    grok {
      match => { \"message\"", :level=>:error}

这样您就可以快速找到您可能遗漏的错误。

logstash gork filter error : fetched an invalid config

logstash gork filter error : fetched an invalid config

logstash

elastic-stack

filebeat