使用 logstash 将日志文件推送到弹性

Question

我们有一个来自 Jmeter 的错误日志文件，我们需要使用 logstash

将其推送到 elastic

这是日志文件，我们想推送第 5 行的数据，

 2021-05-14 20:32:49,822 INFO o.a.j.u.JMeterUtils: Setting Locale to en_EN
2021-05-14 20:32:49,842 INFO o.a.j.JMeter: Loading user properties from: user.properties
2021-05-14 20:32:49,843 INFO o.a.j.JMeter: Loading system properties from: system.properties
2021-05-14 20:32:49,844 WARN o.a.j.JMeter: LogLevel: ERROR
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-4: ; response received: {"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-7: ; response received: {"timestamp":"2021-05-14T14:32:51.689+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-20: ; response received: {"timestamp":"2021-05-14T14:32:51.850+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: LEX 2-1: ; response received: <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-5: ; response received: {"timestamp":"2021-05-14T14:32:51.687+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}

我们希望将第 4 行的数据特别是 PostProcessor: ，收到的响应： 推送到 elastic。以下 conf 仅用于收到的响应：

这是 logstash 配置文件

 if [type] == "errors"{   
     grok {
match => {"message" => "%{GREEDYDATA}#*response received: %{GREEDYDATA:error_message}"}
        }
   }

当我们尝试 push

时出现错误 Grok parse failure _grokparsefailure

不确定我在这里遗漏了什么

Answer 1

^%{GREEDYDATA}ERROR%{GREEDYDATA}response received: %{GREEDYDATA:error_message}

Answer 2

像这样的东西应该适合你：

%{TIMESTAMP_ISO8601:timestamp} %{WORD:LogType} %{GREEDYDATA:data1} PostProcessor: %{GREEDYDATA:data2} ; response received: %{GREEDYDATA:responseRecd}

示例 1：

2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-4: ; response received: {"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}

输出：

{
  "timestamp": [
    [
      "2021-05-14 20:32:51,824"
    ]
  ],
  "LogType": [
    [
      "ERROR"
    ]
  ],
  "data1": [
    [
      "o.a.j.e.J.JSR223"
    ]
  ],
  "data2": [
    [
      "Http URL/API Test 1-4:"
    ]
  ],
  "responseRecd": [
    [
      "{"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}"
    ]
  ]
}

示例 2：

2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: LEX 2-1: ; response received: <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>

输出：

{
  "timestamp": [
    [
      "2021-05-14 20:32:51,824"
    ]
  ],
  "LogType": [
    [
      "ERROR"
    ]
  ],
  "data1": [
    [
      "o.a.j.e.J.JSR223"
    ]
  ],
  "data2": [
    [
      "LEX 2-1:"
    ]
  ],
  "responseRecd": [
    [
      "<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>"
    ]
  ]
}

我已将虚拟名称命名为 data1 和 data2。您可以根据需要进一步将它们分开。请使用 Grok Debugger 进行调试。

使用 logstash 将日志文件推送到弹性

Push log file to elastic using logstash

jmeter

grok

elasticsearch

logstash

logstash-grok