Grok 无法创建可选字段
Grok can't create optional fields
我正在尝试使用 grok 解析 logstash 中的日志。
在错误日志中还有两个可选字段，如 requestUrl 和 requestMethod。
但是当我使用 grok 将错误日志中的日志解析为 logstash.these 时，grok 无法创建这两个字段。
mypipeline.conf :
input {
beats {
type=>"mytest"
port => 5044
}
}
filter{
if [fields][log_type] == "gbase"
{
if [level] in [ "Error", "Fatal" ]
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:timestamp} %{INT:processId} %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{URI:requestUrl} %{USER:requestMethod} %{GREEDYDATA:message}"]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
else
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:timestamp} %{INT:processId} %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{GREEDYDATA:message}" ]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
}
if [fields][log_type] == "finance"
{
if [level] in [ "Error", "Fatal" ]
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:receivedAt} \[%{INT:processId}]\ %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{URI:requestUrl} %{WORD:requestMethod} %{GREEDYDATA:message}"]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
else
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:receivedAt} \[%{INT:processId}]\ %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{GREEDYDATA:message}"]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
#match => [ "time" , "dd/MMM/yyyy:HH:mm:ss Z" ]
target=> "@timestamp"
}
}
output {
if [fields][log_type] == "finance"
{
elasticsearch
{
hosts => ["http://localhost:9200"]
index => "finance-%{+YYYY.MM.dd}"
user => "something"
password => "something"
}
}
if [fields][log_type] == "gbase"
{
elasticsearch
{
hosts => ["http://localhost:9200"]
index => "gbase-%{+YYYY.MM.dd}"
user => "something"
password => "something"
}
}
stdout { codec => rubydebug }
}
filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- D:\Git\gbase.API\API\Logs\*.log
tags: ["gbaseapi"]
fields:
log_type: gbase
multiline.pattern: ^\d\d\d\d-\d\d-\d\d
multiline.negate: true
multiline.match: after
multiline.max_lines : 1000
- type: log
enabled: true
paths:
- D:\Git\finance.api\FinanceAPI\logs\*.log
tags: ["financeapi"]
fields:
log_type: finance
multiline.pattern: ^\d\d\d\d-\d\d-\d\d
multiline.negate: true
multiline.match: after
multiline.max_lines : 1000
我的日志示例：
2020-10-10 21:14:59.8746 22152 WARN Validation bhavin 192.168.43.244 Sensitive data logging is enabled. Log entries and exception messages may include sensitive application data, this mode should only be enabled during development.
2020-10-10 21:15:35.2745 22152 DEBUG QuartzSchedulerThread bhavin 192.168.43.244 Batch acquisition of 0 triggers
2020-10-10 21:15:35.2783 22152 ERROR DeveloperExceptionPageMiddleware bhavin 192.168.43.244 http://bhavin/Leads/New POST An unhandled exception has occurred while executing the request.System.NullReferenceException: Object reference not set to an instance of an object.
json 文档
{
"input" => {
"type" => "log"
},
"processId" => "34076",
"log" => {
"file" => {
"path" => "D:\Git\gbase.API\API\Logs\log-2020-10-10.log"
},
"flags" => [
[0] "multiline"
],
"offset" => 43633
},
"clientIp" => "192.168.43.244",
"fields" => {
"log_type" => "gbase"
},
"timestamp" => "2020-10-10 21:29:00.1492",
"ecs" => {
"version" => "1.5.0"
},
"user" => "bhavin",
"agent" => {
"name" => "Bhavin",
"hostname" => "Bhavin",
"type" => "filebeat",
"version" => "7.9.2",
"id" => "b5decaac-61a3-4dbc-860b-4423a0379391",
"ephemeral_id" => "208d7c9d-bdcd-45e9-8f82-ccdb08292203"
},
"level" => "ERROR",
"message" => "http://bhavin/Leads/New POST An unhandled exception has occurred while executing the request.GurukulERP.Logic.InvalidDataException: Provide a valid PhoneNumber.\n at GurukulERP.API.Controllers.CRM.LeadsController.New(AdmissionLead model) in D:\Git\gbase.API\API\Controllers\CRM\LeadsController.cs:line 515\n at lambda_method(Closure , Object , Object[] )\n at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\n at Microsoft.AspNetCore.Diagnostics..Invoke(HttpContext context) at GurukulERP.API.Controllers.CRM.LeadsController.New(AdmissionLead model) in D:\Git\gbase.API\API\Controllers\CRM\LeadsController.cs:line 515\n at lambda_method(Closure , Object , Object[] )\n at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\n at Microsoft.AspNetCore.Diagnostics..Invoke(HttpContext context)",
"host" => {
"name" => "Bhavin",
"hostname" => "Bhavin",
"architecture" => "x86_64",
"mac" => [
[0] "04:92:26:4a:aa:7b",
[1] "52:3e:aa:6a:59:4c",
[2] "50:3e:aa:6a:59:4c",
[3] "00:ff:73:c0:8e:ca",
[4] "50:3e:aa:6a:59:4c"
],
"id" => "2540a614-ed14-468c-82b6-3c2f1de5b687",
"ip" => [
[ 0] "fe80::8511:1aea:78ea:8722",
[ 1] "169.254.135.34",
[ 2] "fe80::a1dc:1a7d:8369:4ee5",
[ 3] "169.254.78.229",
[ 4] "fe80::c9b5:254f:10b3:191d",
[ 5] "169.254.25.29",
[ 6] "fe80::ac32:76b1:509a:bc5e",
[ 7] "169.254.188.94",
[ 8] "2405:204:808b:8dbb:d456:182e:dd87:ca6",
[ 9] "2405:204:808b:8dbb:3132:508b:53c1:132e",
[10] "fe80::d456:182e:dd87:ca6",
[11] "192.168.43.244"
],
"os" => {
"name" => "Windows 10 Enterprise",
"platform" => "windows",
"kernel" => "10.0.18362.1082 (WinBuild.160101.0800)",
"version" => "10.0",
"family" => "windows",
"build" => "18362.1082"
}
},
"@version" => "1",
"type" => "mytest",
"@timestamp" => 2020-10-10T15:59:02.229Z,
"tags" => [
[0] "gbaseapi",
[1] "beats_input_codec_plain_applied",
[2] "_dateparsefailure"
],
"logger" => "DeveloperExceptionPageMiddleware"
}
这是 kibana 视图的图片：
可以看到其他字段都创建了，但是这两个没有创建
字段 level 在进入您的管道时在您的文档中不存在，因此您在 grok 中使用的条件永远不会匹配 true。
查看您的示例消息，它们具有基本相同的结构，只是在 ip 地址之后开始不同，您可以只使用一个 grok 来解析所有这些消息，然后使用另一个 grok仅当覆盖的 message 字段以 URI 开头时才会应用。
以下过滤器将解析您的所有示例消息。
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{INT:processId} %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{GREEDYDATA:message}" }
overwrite => "message"
}
if [message] =~ "^http:" {
grok {
match => { "message" => "%{URI:requestUrl} %{USER:requestMethod} %{GREEDYDATA:message}" }
overwrite => "message"
}
}
}
第一个 grok 将解析消息并用 IP 地址后的所有内容覆盖 message 字段。因此，对于要提取 requestUrl 和 requestMethod 的消息，您将拥有具有以下值的消息字段。
http://bhavin/Leads/New POST An unhandled exception has occurred while executing the request.System.NullReferenceException: Object reference not set to an instance of an object.
if [message] =~ "^http:" 部分将检查 message 字段是否以 http: 开头，如果 true 它将解析此消息，然后创建字段 requestUrl 和 requestMethod 以及新覆盖的 message 字段。
此外，您的 gsub 过滤器没有意义，您可以将其删除。
您可以将相同的过程应用于您的 finance 日志类型，唯一的区别似乎是 processId 出现在方括号之间，并且您正在使用另一个字段作为时间戳。
我正在尝试使用 grok 解析 logstash 中的日志。 在错误日志中还有两个可选字段，如 requestUrl 和 requestMethod。 但是当我使用 grok 将错误日志中的日志解析为 logstash.these 时，grok 无法创建这两个字段。
mypipeline.conf :
input {
beats {
type=>"mytest"
port => 5044
}
}
filter{
if [fields][log_type] == "gbase"
{
if [level] in [ "Error", "Fatal" ]
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:timestamp} %{INT:processId} %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{URI:requestUrl} %{USER:requestMethod} %{GREEDYDATA:message}"]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
else
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:timestamp} %{INT:processId} %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{GREEDYDATA:message}" ]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
}
if [fields][log_type] == "finance"
{
if [level] in [ "Error", "Fatal" ]
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:receivedAt} \[%{INT:processId}]\ %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{URI:requestUrl} %{WORD:requestMethod} %{GREEDYDATA:message}"]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
else
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:receivedAt} \[%{INT:processId}]\ %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{GREEDYDATA:message}"]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
#match => [ "time" , "dd/MMM/yyyy:HH:mm:ss Z" ]
target=> "@timestamp"
}
}
output {
if [fields][log_type] == "finance"
{
elasticsearch
{
hosts => ["http://localhost:9200"]
index => "finance-%{+YYYY.MM.dd}"
user => "something"
password => "something"
}
}
if [fields][log_type] == "gbase"
{
elasticsearch
{
hosts => ["http://localhost:9200"]
index => "gbase-%{+YYYY.MM.dd}"
user => "something"
password => "something"
}
}
stdout { codec => rubydebug }
}
filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- D:\Git\gbase.API\API\Logs\*.log
tags: ["gbaseapi"]
fields:
log_type: gbase
multiline.pattern: ^\d\d\d\d-\d\d-\d\d
multiline.negate: true
multiline.match: after
multiline.max_lines : 1000
- type: log
enabled: true
paths:
- D:\Git\finance.api\FinanceAPI\logs\*.log
tags: ["financeapi"]
fields:
log_type: finance
multiline.pattern: ^\d\d\d\d-\d\d-\d\d
multiline.negate: true
multiline.match: after
multiline.max_lines : 1000
我的日志示例：
2020-10-10 21:14:59.8746 22152 WARN Validation bhavin 192.168.43.244 Sensitive data logging is enabled. Log entries and exception messages may include sensitive application data, this mode should only be enabled during development.
2020-10-10 21:15:35.2745 22152 DEBUG QuartzSchedulerThread bhavin 192.168.43.244 Batch acquisition of 0 triggers
2020-10-10 21:15:35.2783 22152 ERROR DeveloperExceptionPageMiddleware bhavin 192.168.43.244 http://bhavin/Leads/New POST An unhandled exception has occurred while executing the request.System.NullReferenceException: Object reference not set to an instance of an object.
json 文档
{
"input" => {
"type" => "log"
},
"processId" => "34076",
"log" => {
"file" => {
"path" => "D:\Git\gbase.API\API\Logs\log-2020-10-10.log"
},
"flags" => [
[0] "multiline"
],
"offset" => 43633
},
"clientIp" => "192.168.43.244",
"fields" => {
"log_type" => "gbase"
},
"timestamp" => "2020-10-10 21:29:00.1492",
"ecs" => {
"version" => "1.5.0"
},
"user" => "bhavin",
"agent" => {
"name" => "Bhavin",
"hostname" => "Bhavin",
"type" => "filebeat",
"version" => "7.9.2",
"id" => "b5decaac-61a3-4dbc-860b-4423a0379391",
"ephemeral_id" => "208d7c9d-bdcd-45e9-8f82-ccdb08292203"
},
"level" => "ERROR",
"message" => "http://bhavin/Leads/New POST An unhandled exception has occurred while executing the request.GurukulERP.Logic.InvalidDataException: Provide a valid PhoneNumber.\n at GurukulERP.API.Controllers.CRM.LeadsController.New(AdmissionLead model) in D:\Git\gbase.API\API\Controllers\CRM\LeadsController.cs:line 515\n at lambda_method(Closure , Object , Object[] )\n at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\n at Microsoft.AspNetCore.Diagnostics..Invoke(HttpContext context) at GurukulERP.API.Controllers.CRM.LeadsController.New(AdmissionLead model) in D:\Git\gbase.API\API\Controllers\CRM\LeadsController.cs:line 515\n at lambda_method(Closure , Object , Object[] )\n at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\n at Microsoft.AspNetCore.Diagnostics..Invoke(HttpContext context)",
"host" => {
"name" => "Bhavin",
"hostname" => "Bhavin",
"architecture" => "x86_64",
"mac" => [
[0] "04:92:26:4a:aa:7b",
[1] "52:3e:aa:6a:59:4c",
[2] "50:3e:aa:6a:59:4c",
[3] "00:ff:73:c0:8e:ca",
[4] "50:3e:aa:6a:59:4c"
],
"id" => "2540a614-ed14-468c-82b6-3c2f1de5b687",
"ip" => [
[ 0] "fe80::8511:1aea:78ea:8722",
[ 1] "169.254.135.34",
[ 2] "fe80::a1dc:1a7d:8369:4ee5",
[ 3] "169.254.78.229",
[ 4] "fe80::c9b5:254f:10b3:191d",
[ 5] "169.254.25.29",
[ 6] "fe80::ac32:76b1:509a:bc5e",
[ 7] "169.254.188.94",
[ 8] "2405:204:808b:8dbb:d456:182e:dd87:ca6",
[ 9] "2405:204:808b:8dbb:3132:508b:53c1:132e",
[10] "fe80::d456:182e:dd87:ca6",
[11] "192.168.43.244"
],
"os" => {
"name" => "Windows 10 Enterprise",
"platform" => "windows",
"kernel" => "10.0.18362.1082 (WinBuild.160101.0800)",
"version" => "10.0",
"family" => "windows",
"build" => "18362.1082"
}
},
"@version" => "1",
"type" => "mytest",
"@timestamp" => 2020-10-10T15:59:02.229Z,
"tags" => [
[0] "gbaseapi",
[1] "beats_input_codec_plain_applied",
[2] "_dateparsefailure"
],
"logger" => "DeveloperExceptionPageMiddleware"
}
这是 kibana 视图的图片：
可以看到其他字段都创建了，但是这两个没有创建
字段 level 在进入您的管道时在您的文档中不存在，因此您在 grok 中使用的条件永远不会匹配 true。
查看您的示例消息，它们具有基本相同的结构，只是在 ip 地址之后开始不同，您可以只使用一个 grok 来解析所有这些消息，然后使用另一个 grok仅当覆盖的 message 字段以 URI 开头时才会应用。
以下过滤器将解析您的所有示例消息。
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{INT:processId} %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{GREEDYDATA:message}" }
overwrite => "message"
}
if [message] =~ "^http:" {
grok {
match => { "message" => "%{URI:requestUrl} %{USER:requestMethod} %{GREEDYDATA:message}" }
overwrite => "message"
}
}
}
第一个 grok 将解析消息并用 IP 地址后的所有内容覆盖 message 字段。因此，对于要提取 requestUrl 和 requestMethod 的消息，您将拥有具有以下值的消息字段。
http://bhavin/Leads/New POST An unhandled exception has occurred while executing the request.System.NullReferenceException: Object reference not set to an instance of an object.
if [message] =~ "^http:" 部分将检查 message 字段是否以 http: 开头，如果 true 它将解析此消息，然后创建字段 requestUrl 和 requestMethod 以及新覆盖的 message 字段。
此外，您的 gsub 过滤器没有意义，您可以将其删除。
您可以将相同的过程应用于您的 finance 日志类型，唯一的区别似乎是 processId 出现在方括号之间，并且您正在使用另一个字段作为时间戳。