将匹配器集成到 logstash grok 配置中时出错

Question

我正在使用 logstash 7.6.2。我的日志行是 json 字符串。每个 json 有 3 个字段，"msg" 是文本，"topic" 是文本，"ts" 是浮点数。

这是我的匹配表达式：

{"msg"\s*:\s*(?<msg>".*")\s*,\s*"topic"\s*:\s*(?<topic>".*")\s*,\s*"ts"\s*:\s*(?<ts>[+-]?([0-9]*[.])?[0-9]+)\s*}

以下是两个示例日志行：

{"msg": "2020-05-01 01:09:06,043 ERROR [luna_messaging.handlers.base] HTTP 400: {\"success\": false}\nTraceback (most recent call last):\n  File \"/home/lunalife/luna_messaging/handlers/base.py\", line 238, in wrapper\n    yield func(self, *args, **kwargs)\n  File \"/home/lunalife/.local/lib/python2.7/site-packages/tornado/gen.py\", line 1015, in run\n    value = future.result()\n  File \"/home/lunalife/.local/lib/python2.7/site-packages/tornado/concurrent.py\", line 237, in result\n    raise_exc_info(self._exc_info)\n  File \"/home/lunalife/.local/lib/python2.7/site-packages/tornado/gen.py\", line 1021, in run\n    yielded = self.gen.throw(*exc_info)\n  File \"/home/lunalife/luna_messaging/handlers/device_status.py\", line 41, in get\n    raise tornado.web.HTTPError(400, reason=json.dumps(reason))\nHTTPError: HTTP 400: {\"success\": false}", "topic": "com.walker.prod.luna_messaging.handlers.base", "ts": 1588295346.043578}
{"msg": "2020-05-01 01:09:06,076 ERROR [luna_messaging.handlers.base] HTTP 403: Forbidden\nTraceback (most recent call last):\n  File \"/home/lunalife/luna_messaging/handlers/base.py\", line 238, in wrapper\n    yield func(self, *args, **kwargs)\n  File \"/home/lunalife/.local/lib/python2.7/site-packages/tornado/gen.py\", line 1015, in run\n    value = future.result()\n  File \"/home/lunalife/.local/lib/python2.7/site-packages/tornado/concurrent.py\", line 237, in result\n    raise_exc_info(self._exc_info)\n  File \"/home/lunalife/.local/lib/python2.7/site-packages/tornado/gen.py\", line 1024, in run\n    yielded = self.gen.send(value)\n  File \"/home/lunalife/luna_messaging/handlers/device_status.py\", line 46, in get\n    raise tornado.web.HTTPError(403)\nHTTPError: HTTP 403: Forbidden", "topic": "com.walker.prod.luna_messaging.handlers.base", "ts": 1588295346.076928}```

我使用了几个 grok 测试器来证明它是有效的。 https://grokdebug.herokuapp.com/ and https://grokconstructor.appspot.com/do/match

问题是，当我集成到我的 logstash 配置中时，它给我一个语法错误。我不确定我做错了什么。

这是我的 logstash 配置中的 grok 匹配器：

grok {
   match => {"msg"\s*:\s*(?<msg>".*")\s*,\s*"topic"\s*:\s*(?<topic>".*")\s*,\s*"ts"\s*:\s*(?<ts>[+-]?([0-9]*[.])?[0-9]+)\s*}
}

这是 logstash 启动错误：

Expected one of [ \t\r\n], \"#\", \"=>\" at line 44, column 21

我相信我的匹配表达式是正确的，但我不知道如何将它添加到 grok 配置中。任何帮助将不胜感激。

Answer 1

您需要告诉 grok 过滤器应在哪个字段上应用模式匹配。

正如您从文档 (https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-match) 中看到的那样，匹配设置遵循语法

grok{
  match => { "FIELDNAME" => "PATTERN" }
}

Logstash 将日志行文本放入的默认字段称为 message。所以你会像这样调整你的代码：

grok{
  match => { "message" => "PATTERN" }
}

此外，请注意模式必须被引用并且特殊字符必须被转义（我在下面的例子中没有做后者）。由于您在模式本身中使用了双引号，因此您需要使用单引号，如下所示：

grok{
  match => { 'message' => '{"msg"\s*:\s*(?<msg>".*")\s*,\s*"topic"\s*:\s*(?<topic>".*")\s*,\s*"ts"\s*:\s*(?<ts>[+-]?([0-9]*[.])?[0-9]+)\s*}' }
}

希望能帮到你

将匹配器集成到 logstash grok 配置中时出错

Error integrating matcher into logstash grok config

logstash

logstash-grok

logstash-configuration