是否可以在 Logstash 配置中解析 Caused by of java 日志？

Question

2016-06-02 17:00:32<ERROR><CommonsHTTPTransportSender:361>: 
org.apache.abcd2.abcdFault
    at org.apache.abcd2.abcdFault.makeFault(abcdFault.java:430)
    at org.apache.abcd2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:83)
    at org.apache.abcd2.transport.http.CommonsHTTPTransportSender.sendUsingOutputStream(CommonsHTTPTransportSender.java:358)
    at java.lang.Thread.run(Thread.java:636)
Caused by: com.my.application.IOException: null
    at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1692)
    at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:288)
    at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)
    at org.apache.abcd2.WriteTo(SOAPMessageFormatter.java:79)
    ... 27 more

所以现在如果我想获得像 LOGLEVEL classname 这样的值，并且由我获得了 loglevel 和 class name 的值，但没有得到 Caused by 消息..这怎么可能？

下面是我的配置文件。

input{
    file{
         path => "D:\Log\application.log"
         start_position => beginning

         codec => multiline{
         pattern => "%{TIMESTAMP_ISO8601}"
         what => "next"
         negate => true
        }
    }
}
filter{

        grok{

            match => ["message","^%{TIMESTAMP_ISO8601}<%{LOGLEVEL}><(?<JavaClass>.*[:].*)>"]
        }
        mutate {
        gsub => ['message', "\n", ""]
        gsub => ['message', "\t", ""]
        }
}
output {
        stdout { }
        elasticsearch {
            index => "ABCD_%{+YYYY.MM.dd}"

    }
}

我主要关心的是解析 timestamp loglevel classname 和 causeby values

Answer 1

您的多行编解码器有误，这就是您的配置无法正常工作的原因。（我已经测试过了）。

这是我在我的盒子上使用时你的配置的标准输出（你没有 post）：

artur@pandaadb:~/dev/logstash$ ./logstash-2.3.2/bin/logstash -f conf2/
Settings: Default pipeline workers: 8
Pipeline main started
{
    "@timestamp" => "2016-06-22T09:19:01.896Z",
       "message" => "2016-06-02 17:00:32<ERROR><CommonsHTTPTransportSender:361>: ",
      "@version" => "1",
          "path" => "/home/artur/tmp/logstash/in2/test.log",
          "host" => "pandaadb",
     "JavaClass" => "CommonsHTTPTransportSender:361"
}

看看你的留言怎么连信息都没有？没有什么可以匹配的，因为你的多行不工作。我希望这是第一个问题：

您的模式未反映匹配行的开头
当你否定模式时，你应该做一个 "previous" 而不是下一个。

所以这是一个有效的配置（我使用的是多行过滤器而不是编解码器）：

multiline {
                pattern => "^%{TIMESTAMP_ISO8601}"
                negate => true
                what => "previous"
        }

有了这个，我可以更新你的 grok 以解决问题：

grok {
        match => ["message","^%{TIMESTAMP_ISO8601:ts}<%{LOGLEVEL:log}><(?<JavaClass>.*[:].*)>.*Caused by:%{GREEDYDATA:data}"]
    }

并且运行它与您在我的盒子上的输入，我得到：

artur@pandaadb:~/dev/logstash$ ./logstash-2.3.2/bin/logstash -f conf2/
Settings: Default pipeline workers: 8
Defaulting pipeline worker threads to 1 because there are some filters that might not work with multiple worker threads {:count_was=>8, :filters=>["multiline"], :level=>:warn}
Pipeline main started
{
       "message" => "2016-06-02 17:00:32<ERROR><CommonsHTTPTransportSender:361>: \norg.apache.abcd2.abcdFault\n    at org.apache.abcd2.abcdFault.makeFault(abcdFault.java:430)\n    at org.apache.abcd2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:83)\n    at org.apache.abcd2.transport.http.CommonsHTTPTransportSender.sendUsingOutputStream(CommonsHTTPTransportSender.java:358)\n    at java.lang.Thread.run(Thread.java:636)\nCaused by: com.my.application.IOException: null\n    at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1692)\n    at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:288)\n    at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)\n    at org.apache.abcd2.WriteTo(SOAPMessageFormatter.java:79)\n    ... 27 more",
      "@version" => "1",
    "@timestamp" => "2016-06-22T09:22:38.227Z",
          "path" => "/home/artur/tmp/logstash/in2/test.log",
          "host" => "pandaadb",
          "tags" => [
        [0] "multiline"
    ],
            "ts" => "2016-06-02 17:00:32",
           "log" => "ERROR",
     "JavaClass" => "CommonsHTTPTransportSender:361",
          "data" => " com.my.application.IOException: null\n    at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1692)\n    at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:288)\n    at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)\n    at org.apache.abcd2.WriteTo(SOAPMessageFormatter.java:79)\n    ... 27 more"
}

希望对您有所帮助。

为了将来参考，关于 logstash 问题，举一个使用 stdin 并打印到 stdout 的例子总是有帮助的，因为它可以很快重现。

stdout (rubydebug) 的输出也将告诉您原始消息的确切内容，并使您很容易看出多行不起作用，这就是问题的原因。

干杯！

亚瑟

是否可以在 Logstash 配置中解析 Caused by of java 日志？

Is it possible to get parse of Caused by of java log in Logstash confilg?

logstash

logstash-grok

logstash-configuration

elastic-stack