解析log4j日志的grok
Grok for parsing log4j log
对于下面的grok，我们期望在logstash发送的数据中添加一些字段供kibana分析。但除了消息字段 none 之外，其他字段都可以从 kibana 中看到。
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{SKYLOGLEVEL:loglevel} %{THREAD:thread} %{RMOTEIP:remoteipaddress} %{JAVACLASS:logclass} %{CUSTOM_TRACE_EXCEPTION:exception} %{CUSTOM_TRACE_CAUSED_BY:causedby} %{GREEDYDATA:details}"}
match => {"exception" => "%{CUSTOM_TRACE_EXCEPTION:exception}"}
match => {"thread" => "%{THREAD:thread}"}
match => {"loglevel" => "%{ACMELOGLEVEL:loglevel}"}
match => {"logclass" => "%{JAVACLASS:logclass}"}
match => {"remoteip" => "%{RMOTEIP:remoteipaddress}"}
break_on_match => false
}
任何确定问题的指示都会有所帮助。
谢谢，
三
请从我下面的日志文件中找到一个例外：
2013-04-05 00:00:02,101 ERROR [scheduler_Worker-6 ] (DataProcessor.java:412 ) RemoteException >
AxisFault
faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server
faultSubcode:
faultString: 0005: No Data matched the criteria Specified
faultActor:
faultNode:
faultDetail:
{http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>getNumber</con:node><con:pipeline>getNumber_response</con:pipeline><con:stage>Create Number Response</con:stage><con:path>response-pipeline</con:path></con:location>
0005: No Data matched the criteria Specified1
at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)
at org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)
at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
2013-04-05 00:07:36,535 INFO [TP-Processor8 ] 10.136.59.190 ( WTSDK.java:504 ) WTSDK- Command: V.1
ACDG.WA/ACMEXS/E…/PQF7436
VQZ.
VMF////33080
WM DAH 11417.FAX/BG/RTI/CAM/OZI
2013-04-05 00:07:36,557 INFO [TP-Processor8 ] 10.136.59.190 ( WTSDK.java:505 ) WTSDK- PID: PQF7436
2013-04-05 00:07:40,120 INFO [TP-Processor8 ] 10.136.59.190 ( WTSDK.java:517 ) WTSDK: Response Time before parsing using PID PQF7436 == 3560 ms
2013-04-05 00:07:40,126 INFO [TP-Processor8 ] 10.136.59.190 ( WTSDK.java:547 ) WTSDK: Response string after parsing: WM DAH PERQF11417
摘自 logstash 生成的输出
{"@timestamp":"2016-03-07T23:59:47.306Z","message":"2013-04-05 00:00:02,101 ERROR [scheduler_Worker-6 ] (DataProcessor.java:412 ) RemoteException > \nAxisFault\n faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server\n faultSubcode: \n faultString: 0005: No Data matched the criteria Specified\n faultActor: \n faultNode: \n faultDetail: \n\t{http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>GetNumber</con:node><con:pipeline>GetNumber_response</con:pipeline><con:stage>Create Get Trips By Flight Number Response</con:stage><con:path>response-pipeline</con:path></con:location>\n0005: No Data matched the criteria Specified1\n\tat org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)\n\tat org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)\n\tat org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)\n\tat javax.xml.parsers.SAXParser.parse(Unknown Source)\n\tat org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)\n\tat org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)\n\tat org.apache.axis.Message.getSOAPEnvelope(Message.java:435)\n\tat org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)\n\tat org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)\n\tat org.apache.axis.client.Call.invokeEngine(Call.java:2784)\n\tat org.apache.axis.client.Call.invoke(Call.java:2767)\n\tat org.apache.axis.client.Call.invoke(Call.java:2443)\n\tat org.apache.axis.client.Call.invoke(Call.java:2366)\n\tat org.apache.axis.client.Call.invoke(Call.java:1812)\n\tat com.acme.RequestBindingStub.GetNumber(RequestBindingStub.java:1563)\n\tat com.acme.DataProcessor.callOGSTime(DataProcessor.java:398)\n\tat com.acme.DataProcessor.processData(DataProcessor.java:290)\n\tat sun.reflect.GeneratedMethodAccessor601.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)\n\tat org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)\n\tat org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)\n\tat org.quartz.core.JobRunShell.run(JobRunShell.java:203)\n\tat org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)","@version":"1","tags":["multiline","beats_input_codec_multiline_applied"],"beat":{"hostname":"LVRJ8YRJX1","name":"LVRJ8YRJX1"},"count":1,"fields":null,"input_type":"log","offset":3744,"source":"C:\logs\bagassist_x - Copy.log","type":"log","host":"LVRJ8YRJX1"}
{"@timestamp":"2016-03-07T23:59:47.306Z","message":"2013-04-05 00:00:02,319 ERROR [scheduler_Worker-6 ] (DataProcessor.java:412 ) RemoteException > \nAxisFault\n faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server\n faultSubcode: \n faultString: 0005: No Data matched the criteria Specified\n faultActor: \n faultNode: \n faultDetail: \n\t{http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>GetNumber</con:node><con:pipeline>GetNumber_response</con:pipeline><con:stage>Create Get Trips By Flight Number Response</con:stage><con:path>response-pipeline</con:path></con:location>\n0005: No Data matched the criteria Specified\n\tat org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)\n\tat org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)\n\tat org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)\n\tat javax.xml.parsers.SAXParser.parse(Unknown Source)\n\tat org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)\n\tat org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)\n\tat org.apache.axis.Message.getSOAPEnvelope(Message.java:435)\n\tat org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)\n\tat org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)\n\tat org.apache.axis.client.Call.invokeEngine(Call.java:2784)\n\tat org.apache.axis.client.Call.invoke(Call.java:2767)\n\tat org.apache.axis.client.Call.invoke(Call.java:2443)\n\tat org.apache.axis.client.Call.invoke(Call.java:2366)\n\tat org.apache.axis.client.Call.invoke(Call.java:1812)\n\tat com.acme.RequestBindingStub.GetNumber(RequestBindingStub.java:1563)\n\tat com.acme.DataProcessor.callOGSTime(DataProcessor.java:398)\n\tat com.acme.DataProcessor.processData(DataProcessor.java:290)\n\tat sun.reflect.GeneratedMethodAccessor601.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)\n\tat org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)\n\tat org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)\n\tat org.quartz.core.JobRunShell.run(JobRunShell.java:203)\n\tat org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)","@version":"1","tags":["multiline","beats_input_codec_multiline_applied"],"beat":{"hostname":"LVRJ8YRJX1","name":"LVRJ8YRJX1"},"count":1,"fields":null,"input_type":"log","offset":7569,"source":"C:\logs\bagassist_x - Copy.log","type":"log","host":"LVRJ8YRJX1"}
记录模式
<pattern>%d %-5level [%-28thread] [%-15X{remoteIpAddress}] (%35logger{0}:%-3L\) %message%n</pattern>
为了分析log4j日志，你可以使用下面的grok过滤器：
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %
{SKYLOGLEVEL:loglevel} %{THREAD:thread} %{RMOTEIP:remoteipaddress} %
{JAVACLASS:logclass} %{CUSTOM_TRACE_EXCEPTION:exception} %
{CUSTOM_TRACE_CAUSED_BY:causedby} %{GREEDYDATA:details}"
}
在上面的 grok 过滤器中，您可以过滤整个 log4j 日志。
您可以使用这个 gork 构造函数，然后根据您的要求进行自定义，它非常简单易用：http://grokconstructor.appspot.com/do/translator
这是我的模式：
- Input: 09-05-18 10:40:57,384 43 11296 [Timer-3] INFO abc.com.task.CheckPendingTask - ---- process START!!! ----
- Pattern: %{TIMESTAMP_ISO8601:timestamp} %{NUMBER:line} %{NUMBER:relativeTime} %{NOTSPACE:thread} %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:message}
- Output:
{
"timestamp": [
[
"09-05-18 10:40:57,384"
]
],
"YEAR": [
[
"09"
]
],
"MONTHNUM": [
[
"05"
]
],
"MONTHDAY": [
[
"18"
]
],
"HOUR": [
[
"10",
null
]
],
"MINUTE": [
[
"40",
null
]
],
"SECOND": [
[
"57,384"
]
],
"ISO8601_TIMEZONE": [
[
null
]
],
"line": [
[
"43"
]
],
"BASE10NUM": [
[
"43",
"11296"
]
],
"relativeTime": [
[
"11296"
]
],
"thread": [
[
"[Timer-3]"
]
],
"loglevel": [
[
"INFO"
]
],
"class": [
[
"abc.com.task.CheckPendingTask"
]
],
"message": [
[
" ---- process START!!! ---- "
]
]
}
对于下面的grok，我们期望在logstash发送的数据中添加一些字段供kibana分析。但除了消息字段 none 之外，其他字段都可以从 kibana 中看到。
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{SKYLOGLEVEL:loglevel} %{THREAD:thread} %{RMOTEIP:remoteipaddress} %{JAVACLASS:logclass} %{CUSTOM_TRACE_EXCEPTION:exception} %{CUSTOM_TRACE_CAUSED_BY:causedby} %{GREEDYDATA:details}"}
match => {"exception" => "%{CUSTOM_TRACE_EXCEPTION:exception}"}
match => {"thread" => "%{THREAD:thread}"}
match => {"loglevel" => "%{ACMELOGLEVEL:loglevel}"}
match => {"logclass" => "%{JAVACLASS:logclass}"}
match => {"remoteip" => "%{RMOTEIP:remoteipaddress}"}
break_on_match => false
}
任何确定问题的指示都会有所帮助。
谢谢，
三
请从我下面的日志文件中找到一个例外：
2013-04-05 00:00:02,101 ERROR [scheduler_Worker-6 ] (DataProcessor.java:412 ) RemoteException >
AxisFault
faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server
faultSubcode:
faultString: 0005: No Data matched the criteria Specified
faultActor:
faultNode:
faultDetail:
{http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>getNumber</con:node><con:pipeline>getNumber_response</con:pipeline><con:stage>Create Number Response</con:stage><con:path>response-pipeline</con:path></con:location>
0005: No Data matched the criteria Specified1
at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)
at org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)
at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
2013-04-05 00:07:36,535 INFO [TP-Processor8 ] 10.136.59.190 ( WTSDK.java:504 ) WTSDK- Command: V.1
ACDG.WA/ACMEXS/E…/PQF7436
VQZ.
VMF////33080
WM DAH 11417.FAX/BG/RTI/CAM/OZI
2013-04-05 00:07:36,557 INFO [TP-Processor8 ] 10.136.59.190 ( WTSDK.java:505 ) WTSDK- PID: PQF7436
2013-04-05 00:07:40,120 INFO [TP-Processor8 ] 10.136.59.190 ( WTSDK.java:517 ) WTSDK: Response Time before parsing using PID PQF7436 == 3560 ms
2013-04-05 00:07:40,126 INFO [TP-Processor8 ] 10.136.59.190 ( WTSDK.java:547 ) WTSDK: Response string after parsing: WM DAH PERQF11417
摘自 logstash 生成的输出
{"@timestamp":"2016-03-07T23:59:47.306Z","message":"2013-04-05 00:00:02,101 ERROR [scheduler_Worker-6 ] (DataProcessor.java:412 ) RemoteException > \nAxisFault\n faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server\n faultSubcode: \n faultString: 0005: No Data matched the criteria Specified\n faultActor: \n faultNode: \n faultDetail: \n\t{http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>GetNumber</con:node><con:pipeline>GetNumber_response</con:pipeline><con:stage>Create Get Trips By Flight Number Response</con:stage><con:path>response-pipeline</con:path></con:location>\n0005: No Data matched the criteria Specified1\n\tat org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)\n\tat org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)\n\tat org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)\n\tat javax.xml.parsers.SAXParser.parse(Unknown Source)\n\tat org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)\n\tat org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)\n\tat org.apache.axis.Message.getSOAPEnvelope(Message.java:435)\n\tat org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)\n\tat org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)\n\tat org.apache.axis.client.Call.invokeEngine(Call.java:2784)\n\tat org.apache.axis.client.Call.invoke(Call.java:2767)\n\tat org.apache.axis.client.Call.invoke(Call.java:2443)\n\tat org.apache.axis.client.Call.invoke(Call.java:2366)\n\tat org.apache.axis.client.Call.invoke(Call.java:1812)\n\tat com.acme.RequestBindingStub.GetNumber(RequestBindingStub.java:1563)\n\tat com.acme.DataProcessor.callOGSTime(DataProcessor.java:398)\n\tat com.acme.DataProcessor.processData(DataProcessor.java:290)\n\tat sun.reflect.GeneratedMethodAccessor601.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)\n\tat org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)\n\tat org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)\n\tat org.quartz.core.JobRunShell.run(JobRunShell.java:203)\n\tat org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)","@version":"1","tags":["multiline","beats_input_codec_multiline_applied"],"beat":{"hostname":"LVRJ8YRJX1","name":"LVRJ8YRJX1"},"count":1,"fields":null,"input_type":"log","offset":3744,"source":"C:\logs\bagassist_x - Copy.log","type":"log","host":"LVRJ8YRJX1"}
{"@timestamp":"2016-03-07T23:59:47.306Z","message":"2013-04-05 00:00:02,319 ERROR [scheduler_Worker-6 ] (DataProcessor.java:412 ) RemoteException > \nAxisFault\n faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server\n faultSubcode: \n faultString: 0005: No Data matched the criteria Specified\n faultActor: \n faultNode: \n faultDetail: \n\t{http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>GetNumber</con:node><con:pipeline>GetNumber_response</con:pipeline><con:stage>Create Get Trips By Flight Number Response</con:stage><con:path>response-pipeline</con:path></con:location>\n0005: No Data matched the criteria Specified\n\tat org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)\n\tat org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)\n\tat org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)\n\tat javax.xml.parsers.SAXParser.parse(Unknown Source)\n\tat org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)\n\tat org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)\n\tat org.apache.axis.Message.getSOAPEnvelope(Message.java:435)\n\tat org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)\n\tat org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)\n\tat org.apache.axis.client.Call.invokeEngine(Call.java:2784)\n\tat org.apache.axis.client.Call.invoke(Call.java:2767)\n\tat org.apache.axis.client.Call.invoke(Call.java:2443)\n\tat org.apache.axis.client.Call.invoke(Call.java:2366)\n\tat org.apache.axis.client.Call.invoke(Call.java:1812)\n\tat com.acme.RequestBindingStub.GetNumber(RequestBindingStub.java:1563)\n\tat com.acme.DataProcessor.callOGSTime(DataProcessor.java:398)\n\tat com.acme.DataProcessor.processData(DataProcessor.java:290)\n\tat sun.reflect.GeneratedMethodAccessor601.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)\n\tat org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)\n\tat org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)\n\tat org.quartz.core.JobRunShell.run(JobRunShell.java:203)\n\tat org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)","@version":"1","tags":["multiline","beats_input_codec_multiline_applied"],"beat":{"hostname":"LVRJ8YRJX1","name":"LVRJ8YRJX1"},"count":1,"fields":null,"input_type":"log","offset":7569,"source":"C:\logs\bagassist_x - Copy.log","type":"log","host":"LVRJ8YRJX1"}
记录模式
<pattern>%d %-5level [%-28thread] [%-15X{remoteIpAddress}] (%35logger{0}:%-3L\) %message%n</pattern>
为了分析log4j日志，你可以使用下面的grok过滤器：
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %
{SKYLOGLEVEL:loglevel} %{THREAD:thread} %{RMOTEIP:remoteipaddress} %
{JAVACLASS:logclass} %{CUSTOM_TRACE_EXCEPTION:exception} %
{CUSTOM_TRACE_CAUSED_BY:causedby} %{GREEDYDATA:details}"
}
在上面的 grok 过滤器中，您可以过滤整个 log4j 日志。
您可以使用这个 gork 构造函数，然后根据您的要求进行自定义，它非常简单易用：http://grokconstructor.appspot.com/do/translator
这是我的模式：
- Input: 09-05-18 10:40:57,384 43 11296 [Timer-3] INFO abc.com.task.CheckPendingTask - ---- process START!!! ----
- Pattern: %{TIMESTAMP_ISO8601:timestamp} %{NUMBER:line} %{NUMBER:relativeTime} %{NOTSPACE:thread} %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:message}
- Output:
{
"timestamp": [
[
"09-05-18 10:40:57,384"
]
],
"YEAR": [
[
"09"
]
],
"MONTHNUM": [
[
"05"
]
],
"MONTHDAY": [
[
"18"
]
],
"HOUR": [
[
"10",
null
]
],
"MINUTE": [
[
"40",
null
]
],
"SECOND": [
[
"57,384"
]
],
"ISO8601_TIMEZONE": [
[
null
]
],
"line": [
[
"43"
]
],
"BASE10NUM": [
[
"43",
"11296"
]
],
"relativeTime": [
[
"11296"
]
],
"thread": [
[
"[Timer-3]"
]
],
"loglevel": [
[
"INFO"
]
],
"class": [
[
"abc.com.task.CheckPendingTask"
]
],
"message": [
[
" ---- process START!!! ---- "
]
]
}