websphere/java 日志的 Logstash 多行过滤器

Logstash Multiline filter for websphere/java logs

您好,我的 logstash 多行配置有问题。我正在解析 websphere/java 日志,并且多行在某些情况下不起作用。

我的多行配置如下所示。我尝试了几种类型的正则表达式,但没有一种有效。

 codec => multiline {
     pattern => "^\A%{SYSLOG5424SD}"
     negate => true
     what => previous
 }

这是未正确解析的日志示例:

[1.6.2015 15:02:46:635 CEST] 00000109 BusinessExcep E   CNTR0020E: EJB threw an unexpected (non-declared) exception during invocation of method     "processCommand" on bean     "BeanId(Issz_Produkcia_2.1.63#Ssz_Server_EJB.jar#CommandDispatcherImpl, null)".   Exception data: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back; nested exception is: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back; nested exception is: javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout
javax.ejb.EJBTransactionRolledbackException: Transaction rolled back; nested exception is: javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout
javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout
at com.ibm.tx.jta.impl.EmbeddableTranManagerImpl.completeTxTimeout(EmbeddableTranManagerImpl.java:62)
at com.ibm.tx.jta.impl.EmbeddableTranManagerSet.completeTxTimeout(EmbeddableTranManagerSet.java:85)
at com.ibm.ejs.csi.TransactionControlImpl.completeTxTimeout(TransactionControlImpl.java:1347)
at com.ibm.ejs.csi.TranStrategy.postInvoke(TranStrategy.java:273)
at com.ibm.ejs.csi.TransactionControlImpl.postInvoke(TransactionControlImpl.java:579)
at com.ibm.ejs.container.EJSContainer.postInvoke(EJSContainer.java:4874)
at sk.sits.upsvar.server.ejb.entitymanagers.EJSLocal0SLDokumentManagerImpl_18dd4eb4.findAllDokumentPripadByCriteriaMap(EJSLocal0SLDokumentManagerImpl_18dd4eb4.java)
at sk.sits.upsvar.server.ejb.DataAccessServiceImpl.executeDokumentCmd(DataAccessServiceImpl.java:621)
at sk.sits.upsvar.server.ejb.DataAccessServiceImpl.executeCmd(DataAccessServiceImpl.java:220)
at sk.sits.upsvar.server.ejb.EJSLocal0SLDataAccessServiceImpl_6e5b0656.executeCmd(EJSLocal0SLDataAccessServiceImpl_6e5b0656.java)
at sk.sits.upsvar.server.ejb.CommandDispatcherImpl.processSoloCommand(CommandDispatcherImpl.java:222)
at sk.sits.upsvar.server.ejb.CommandDispatcherImpl._processCommand(CommandDispatcherImpl.java:151)
at sk.sits.upsvar.server.ejb.CommandDispatcherImpl.processCommand(CommandDispatcherImpl.java:100)
at sk.sits.upsvar.server.ejb.EJSLocal0SLCommandDispatcherImpl_b974dd5c.processCommand(EJSLocal0SLCommandDispatcherImpl_b974dd5c.java)
at sk.sits.upsvar.server.ejb.SszServiceImpl.process(SszServiceImpl.java:146)
at sk.sits.upsvar.server.ejb.EJSRemote0SLSszService_8e2ee81c.process(EJSRemote0SLSszService_8e2ee81c.java)
at sk.sits.upsvar.server.ejb._EJSRemote0SLSszService_8e2ee81c_Tie.process(_EJSRemote0SLSszService_8e2ee81c_Tie.java)
at sk.sits.upsvar.server.ejb._EJSRemote0SLSszService_8e2ee81c_Tie._invoke(_EJSRemote0SLSszService_8e2ee81c_Tie.java)
at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:678)
at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:525)
at com.ibm.rmi.iiop.ORB.process(ORB.java:576)
at com.ibm.CORBA.iiop.ORB.process(ORB.java:1578)
at com.ibm.rmi.iiop.Connection.doRequestWork(Connection.java:3076)
at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2946)
at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:64)
at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1700)
javax.ejb.EJBTransactionRolledbackException: Transaction rolled back; nested exception is: javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout
 Caused by: javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout
at com.ibm.tx.jta.impl.EmbeddableTranManagerImpl.completeTxTimeout(EmbeddableTranManagerImpl.java:62)
at com.ibm.tx.jta.impl.EmbeddableTranManagerSet.completeTxTimeout(EmbeddableTranManagerSet.java:85)
at com.ibm.ejs.csi.TransactionControlImpl.completeTxTimeout(TransactionControlImpl.java:1347)
at com.ibm.ejs.csi.TranStrategy.postInvoke(TranStrategy.java:273)
at com.ibm.ejs.csi.TransactionControlImpl.postInvoke(TransactionControlImpl.java:579)
at com.ibm.ejs.container.EJSContainer.postInvoke(EJSContainer.java:4874)
at sk.sits.upsvar.server.ejb.entitymanagers.EJSLocal0SLDokumentManagerImpl_18dd4eb4.findAllDokumentPripadByCriteriaMap(EJSLocal0SLDokumentManagerImpl_18dd4eb4.java)
at sk.sits.upsvar.server.ejb.DataAccessServiceImpl.executeDokumentCmd(DataAccessServiceImpl.java:621)
at sk.sits.upsvar.server.ejb.DataAccessServiceImpl.executeCmd(DataAccessServiceImpl.java:220)
at sk.sits.upsvar.server.ejb.EJSLocal0SLDataAccessServiceImpl_6e5b0656.executeCmd(EJSLocal0SLDataAccessServiceImpl_6e5b0656.java)
at sk.sits.upsvar.server.ejb.CommandDispatcherImpl.processSoloCommand(CommandDispatcherImpl.java:222)
at sk.sits.upsvar.server.ejb.CommandDispatcherImpl._processCommand(CommandDispatcherImpl.java:151)
at sk.sits.upsvar.server.ejb.CommandDispatcherImpl.processCommand(CommandDispatcherImpl.java:100)
at sk.sits.upsvar.server.ejb.EJSLocal0SLCommandDispatcherImpl_b974dd5c.processCommand(EJSLocal0SLCommandDispatcherImpl_b974dd5c.java)
at sk.sits.upsvar.server.ejb.SszServiceImpl.process(SszServiceImpl.java:146)
at sk.sits.upsvar.server.ejb.EJSRemote0SLSszService_8e2ee81c.process(EJSRemote0SLSszService_8e2ee81c.java)
at sk.sits.upsvar.server.ejb._EJSRemote0SLSszService_8e2ee81c_Tie.process(_EJSRemote0SLSszService_8e2ee81c_Tie.java)
at sk.sits.upsvar.server.ejb._EJSRemote0SLSszService_8e2ee81c_Tie._invoke(_EJSRemote0SLSszService_8e2ee81c_Tie.java)
at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:678)
at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:525)
at com.ibm.rmi.iiop.ORB.process(ORB.java:576)
at com.ibm.CORBA.iiop.ORB.process(ORB.java:1578)
at com.ibm.rmi.iiop.Connection.doRequestWork(Connection.java:3076)
at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2946)
at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:64)
at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1700)

它是按行解析的,我需要将它一起解析。不知道是不是有什么性格把他们分开了

我尝试了这些模式:

pattern => "%{DATESTAMP} %{WORD:zone}]"
pattern => "^\["
pattern => "\A"

还有很多我不记得了。遇到过这个问题的人可以帮助我吗。

非常感谢。

这是我的完整配置。

input {
    file {  
        path => "D:\Log\Logstash\testlog.log"
        type => "LOG"
        start_position => "beginning"
        codec => plain {
            charset => "ISO-8859-1"
        }
        codec => multiline {
            pattern => "^\A%{SYSLOG5424SD}"
            negate => true
             what => previous
        }
    } 
}
filter {
    grok{
        match => [ "message",".*exception.*"]
        add_tag => "exception"
    }
    mutate{
        remove_tag => "_grokparsefailure"
    }
    grok    {
        match => [ "message","%{DATESTAMP} %{WORD:}] %{WORD:} %{WORD:}\s* W"]
        add_tag => "Warning"
        remove_tag => "_grokparsefailure"
    }
    grok    {
        match => [ "message","%{DATESTAMP} %{WORD:}] %{WORD:} %{WORD:}\s* F"]
        add_tag => "Fatal"
        remove_tag => "_grokparsefailure"
    }
    grok    {
        match => [ "message","%{DATESTAMP} %{WORD:}] %{WORD:} %{WORD:}\s* O"]
        add_tag => "Message"
        remove_tag => "_grokparsefailure"
    }
    grok    {
        match => [ "message","%{DATESTAMP} %{WORD:}] %{WORD:} %{WORD:}\s* C"]
        add_tag => "Config"
        remove_tag => "_grokparsefailure"
    }
    #if ("Warning" not in [tags]) {
        grok    {
            match => [ "message","%{DATESTAMP} %{WORD:}] %{WORD:} %{WORD:}\s* E"]
            add_tag => "Error"
            remove_tag => "_grokparsefailure"
        }
    #}else {
        grok    {
            match => [ "message","%{DATESTAMP} %{WORD:}] %{WORD:} %{WORD:   }\s* I"]
            add_tag => "Info"
        }
    #}
    grok {
        match => [ "message", "%{DATESTAMP} %{WORD:zone}] %{WORD:ID} %{WORD:CLASS}\s* . (.*\s){0,}%{GREEDYDATA:OBSAH}" ]
    remove_tag => "_grokparsefailure"
    }
    grok {
        match => [ "message", "%{DATESTAMP} %{WORD:zone}] %{WORD:ID} %{WORD:CLASS}\s* . (.*\s){0,}%{WORD:WAS_CODE}:%{GREEDYDATA:OBSAH}" ]
                                            #"message","%{DATESTAMP} %{WORD:zone}] %{WORD:ID} %{WORD:CLASS}\s* W \s*\[SID:%{WORD:ISSZSID}]%{GREEDYDATA:OBSAH}"]
        remove_tag => "_grokparsefailure"
        add_tag => "was_error"
    }
    if ("was_error" not in [tags]) { 
        grok {
            match => [ "message","%{DATESTAMP} %{WORD:zone}] %{WORD:ID} %{WORD:CLASS}\s* . \s*\[SID:%{WORD:ISSZSID}]%{GREEDYDATA:OBSAH}" ]
            remove_tag => "_grokparsefailure"
        }
        if "_grokparsefailure" not in [tags] {
            if [ISSZSID] != "null" {
                mutate{ 
                    add_tag => "ISSZwithID"
                    remove_tag => "_grokparsefailure"
                }
            } else {
                mutate{
                    add_tag => "ISSZnull"
                    remove_tag => "_grokparsefailure"
                }
            }
        }
    }
}

output {
    if "_grokparsefailure" not in [tags] {
        elasticsearch {
            hosts => ["127.0.0.1:9200"]
            #protocol => "http"
        }
    }
    stdout {}
}

假设使用 multiline 作为编解码器和另一个编解码器并不是它的预期目的。我宁愿将它用作单个编解码器或过滤器。

将你的配置转换成这个,你会得到你想要的结果:

input {
  file {    
    path => "D:\Log\Logstash\testlog.log"
    type => "LOG"
    start_position => "beginning"
    codec => plain { charset => "ISO-8859-1" }
  } 
}
filter {
    multiline {
        pattern => "^\A%{SYSLOG5424SD}"
        negate => true
        what => previous
    }
    # ... all other filters 
}
output {
 # your output definitions
}

一个著名的多行解析示例是来自 Jordan Sissle 的 MySQL 日志解析示例:https://gist.github.com/jordansissel/3753353

干杯