logstash: wso api 管理器记录到弹性

Question

我有如下日志，日志中的每个参数都是静态的。我尝试通过 logstash 发送弹性并使用 Grok pattern.But 我看不到常规日志。我不擅长神交。我如何编写合适的 grok 代码？在输出中我不想看到年份、月份信息和空的 GREEDYDATA 部分。

[2021-12-05 20:57:40,513]  INFO - LogCounterMetric Metric Name : apim:response Metric Value: {apiName=ApiExp, proxyResponseCode=200, errorType=null, destination=http://1.1.1.1:11/service.asmx, apiCreatorTenantDomain=carbon.super, platform=Other, apiMethod=POST, apiVersion=1.0, gatewayType=SYNAPSE, apiCreator=admin, responseCacheHit=false, backendLatency=229, correlationId=244a4509-52a6-4f37-921d-5d2fc6e40ca3, requestMediationLatency=365, keyType=SANDBOX, apiId=32154b48-70dc5-4cx5-9dgd-7er32e31bacc, applicationName=DefaultApplication, targetResponseCode=200, requestTimestamp=2021-12-05T20:57:39.823Z, applicationOwner=admin, userAgent=Other, eventType=response, apiResourceTemplate=/*, responseLatency=596, regionId=default, responseMediationLatency=2, userIp=1.1.1.1, applicationId=37d453cc-4425-fd31-gf9c-fcdfae89bbd, apiType=SOAP}

这是我写的grok代码。

\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}%{LOGLEVEL:level} - LogCounterMetric Metric Name : apim:response Metric Value: {%{GREEDYDATA}apiName=%{GREEDYDATA:apiName}, %{GREEDYDATA}proxyResponseCode=%{GREEDYDATA:proxyResponseCode}, %{GREEDYDATA}errorType=%{GREEDYDATA:errorType}, %{GREEDYDATA}destination=%{GREEDYDATA:destination}, %{GREEDYDATA}apiCreatorTenantDomain=%{GREEDYDATA:apiCreatorTenantDomain}, %{GREEDYDATA}platform=%{GREEDYDATA:platform}, %{GREEDYDATA}apiMethod=%{GREEDYDATA:apiMethod}, %{GREEDYDATA}apiVersion=%{GREEDYDATA:apiVersion}, %{GREEDYDATA}gatewayType=%{GREEDYDATA:gatewayType}, %{GREEDYDATA}apiCreator=%{GREEDYDATA:apiCreator}, %{GREEDYDATA}responseCacheHit=%{GREEDYDATA:responseCacheHit}, %{GREEDYDATA}backendLatency=%{GREEDYDATA:backendLatency}, %{GREEDYDATA}correlationId=%{GREEDYDATA:correlationId, %{GREEDYDATA}requestMediationLatency=%{GREEDYDATA:requestMediationLatency}, %{GREEDYDATA}keyType=%{GREEDYDATA:keyType}, %{GREEDYDATA}apiId=%{GREEDYDATA:apiId}, %{GREEDYDATA}applicationName=%{GREEDYDATA:applicationName}, %{GREEDYDATA}targetResponseCode=%{GREEDYDATA:targetResponseCode}, %{GREEDYDATA}requestTimestamp=%{GREEDYDATA:requestTimestamp}, %{GREEDYDATA}applicationOwner=%{GREEDYDATA:applicationOwner}, %{GREEDYDATA}userAgent=%{GREEDYDATA:userAgent}, %{GREEDYDATA}eventType=%{GREEDYDATA:eventType}, %{GREEDYDATA}apiResourceTemplate=%{GREEDYDATA:apiResourceTemplate}, %{GREEDYDATA}responseLatency=%{GREEDYDATA:responseLatency}, %{GREEDYDATA}regionId=%{GREEDYDATA:regionId}, %{GREEDYDATA}responseMediationLatency=%{GREEDYDATA:responseMediationLatency}, %{GREEDYDATA}userIp=%{GREEDYDATA:userIp}, %{GREEDYDATA}applicationId=%{GREEDYDATA:applicationId}, %{GREEDYDATA}apiType=%{GREEDYDATA:apiType}}

--grok调试器输出

{
  "timestamp": [
    [
      "2021-12-05 20:57:40,513"
    ]
  ],
  "YEAR": [
    [
      "2021"
    ]
  ],
  "MONTHNUM": [
    [
      "12"
    ]
  ],
  "MONTHDAY": [
    [
      "05"
    ]
  ],
  "HOUR": [
    [
      "20",
      null
    ]
  ],
  "MINUTE": [
    [
      "57",
      null
    ]
  ],
  "SECOND": [
    [
      "40,513"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "SPACE": [
    [
      "  "
    ]
  ],
  "level": [
    [
      "INFO"
    ]
  ],
  "GREEDYDATA": [
    [
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      ""
    ]
  ],
  "apiName": [
    [
      "ApiExp"
    ]
  ],
  "proxyResponseCode": [
    [
      "200"
    ]
  ],
  "errorType": [
    [
      "null"
    ]
  ],
  "destination": [
    [
      "http://1.1.1.1:11/service.asmx"
    ]
  ],
  "apiCreatorTenantDomain": [
    [
      "carbon.super"
    ]
  ],
  "platform": [
    [
      "Other"
    ]
  ],
  "apiMethod": [
    [
      "POST"
    ]
  ],
  "apiVersion": [
    [
      "1.0"
    ]
  ],
  "gatewayType": [
    [
      "SYNAPSE"
    ]
  ],
  "apiCreator": [
    [
      "admin"
    ]
  ],
  "responseCacheHit": [
    [
      "false"
    ]
  ],
  "backendLatency": [
    [
      "229"
    ]
  ],
  "correlationId": [
    [
      "244a4509-52a6-4f37-921d-5d2fc6e40ca3, "
    ]
  ],
  "requestMediationLatency": [
    [
      "365"
    ]
  ],
  "keyType": [
    [
      "SANDBOX"
    ]
  ],
  "apiId": [
    [
      "32154b48-70dc5-4cx5-9dgd-7er32e31bacc"
    ]
  ],
  "applicationName": [
    [
      "DefaultApplication"
    ]
  ],
  "targetResponseCode": [
    [
      "200"
    ]
  ],
  "requestTimestamp": [
    [
      "2021-12-05T20:57:39.823Z"
    ]
  ],
  "applicationOwner": [
    [
      "admin"
    ]
  ],
  "userAgent": [
    [
      "Other"
    ]
  ],
  "eventType": [
    [
      "response"
    ]
  ],
  "apiResourceTemplate": [
    [
      "/*"
    ]
  ],
  "responseLatency": [
    [
      "596"
    ]
  ],
  "regionId": [
    [
      "default"
    ]
  ],
  "responseMediationLatency": [
    [
      "2"
    ]
  ],
  "userIp": [
    [
      "1.1.1.1"
    ]
  ],
  "applicationId": [
    [
      "37d453cc-4425-fd31-gf9c-fcdfae89bbd"
    ]
  ],
  "apiType": [
    [
      "SOAP"
    ]
  ]
}

Answer 1

您可以使用 删除过滤器 的选项 remove_field 来删除输出中不需要的特定字段。

更多详情：

https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html#plugins-filters-drop-remove_field

logstash: wso api 管理器记录到弹性

logstash: wso api manager logs to elastic

wso2

grok

elasticsearch

logstash-grok