使用filebeat将结构化日志数据直接推送到elasticsearch

Question

我已将 filebeat 配置为收集我的结构化日志输出（绿地项目，因此每个日志条目都是预定义格式的 JSON 文档）并将其直接发布到 ELS。

示例日志文件摘录（请注意 additional 是自由形式，所有其他属性都是固定的。此 post 的格式很漂亮，但每个顶级对象在文件中都位于一行中） :

{
    "TimeUtc": "2016-09-23T14:13:02.217520245Z",
    "ServiceKey": "MAAS_SVC",
    "Title": "Get All Campaigns - Start",
    "Additional": {
        "HTTPRequest": {
            "Method": "GET",
            "URL": {
                "Scheme": "",
                "Opaque": "",
                "User": null,
                "Host": "",
                "Path": "/admin/campaigns",
                "RawPath": "",
                "ForceQuery": false,
                "RawQuery": "",
                "Fragment": ""
            },
            "Proto": "HTTP/1.1",
            "ProtoMajor": 1,
            "ProtoMinor": 1,
            "Header": {
                "Accept": ["*/*"],
                "Accept-Encoding": ["gzip, deflate"],
                "Connection": ["keep-alive"],
                "Requestkey": ["78478050-47f0-4d0d-44e8-615d0599574a"],
                "User-Agent": ["python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic"]
            },
            "Body": {
                "Closer": {
                    "Reader": null
                }
            },
            "ContentLength": 0,
            "TransferEncoding": null,
            "Close": false,
            "Host": "xxxxxxxxx",
            "Form": null,
            "PostForm": null,
            "MultipartForm": null,
            "Trailer": null,
            "RemoteAddr": "xxx.xxx.xxx.xxx",
            "RequestURI": "/admin/campaigns",
            "TLS": null,
            "Cancel": ,
            "Response": null
        }
    },
    "RequestKey": "78478050-47f0-4d0d-44e8-615d0599574a",
    "HostAddress": "xxxxxxxxx"
}

这导致 filebeat 向 ELS 发出以下请求：

{
    "@timestamp": "2016-10-12T13:53:21.597Z",
    "beat": {
        "hostname": "7bca0e28e69e",
        "name": "7bca0e28e69e"
    },
    "count": 1,
    "fields": null,
    "input_type": "log",
    "message": "{\"TimeUtc\":\"2016-09-23T14:13:02.217520245Z\",\"ServiceKey\":\"MAAS_SVC\",\"Title\":\"Get All Campaigns - Start\",\"Additional\":{\"HTTPRequest\":{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/admin/campaigns\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Connection\":[\"keep-alive\"],\"Requestkey\":[\"78478050-47f0-4d0d-44e8-615d0599574a\"],\"User-Agent\":[\"python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic\"]},\"Body\":{\"Closer\":{\"Reader\":null}},\"ContentLength\":0,\"TransferEncoding\":null,\"Close\":false,\"Host\":\"bistromath.marathon.mesos:40072\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.1.70:42854\",\"RequestURI\":\"/admin/campaigns\",\"TLS\":null,\"Cancel\":,\"Response\":null}},\"RequestKey\":\"78478050-47f0-4d0d-44e8-615d0599574a\",\"HostAddress\":\"ba47316c9c45\"}",
    "offset": 0,
    "source": "/filebeat/log-harvest/maas-service-single.log",
    "type": "log"
}

我能否阻止 filebeat 转义我的日志 JSON 以便它成为嵌套对象而不是字符串，或者我是否需要修补 filebeat？

Answer 1

可以在 Filebeat 5.x 中解析 JSON 消息，但不能在 Filebeat 1.x 中解析。可以在配置文件中指定 json 选项。

如果您仅限于使用 Filebeat 1.x，那么您将需要 Logstash 来解析来自 message 字段的 JSON 数据。您将配置 Filebeat -> Logstash -> Elasticsearch。

Filebeat 5.x 配置：

filebeat:
  prospectors:
    - paths:
        - input.json
      json.message_key: Title
      json.keys_under_root: true
      json.add_error_key: true

output:
  console:
    pretty: true

示例输出：

{
  "@timestamp": "2016-10-12T22:40:16.338Z",
  "Additional": {
    "HTTPRequest": {
      "Body": {
        "Closer": {}
      },
      "Close": false,
      "ContentLength": 0,
      "Header": {
        "Accept": [
          "*/*"
        ],
        "Accept-Encoding": [
          "gzip, deflate"
        ],
        "Connection": [
          "keep-alive"
        ],
        "Requestkey": [
          "78478050-47f0-4d0d-44e8-615d0599574a"
        ],
        "User-Agent": [
          "python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic"
        ]
      },
      "Host": "xxxxxxxxx",
      "Method": "GET",
      "Proto": "HTTP/1.1",
      "ProtoMajor": 1,
      "ProtoMinor": 1,
      "RemoteAddr": "xxx.xxx.xxx.xxx",
      "RequestURI": "/admin/campaigns",
      "URL": {
        "ForceQuery": false,
        "Fragment": "",
        "Host": "",
        "Opaque": "",
        "Path": "/admin/campaigns",
        "RawPath": "",
        "RawQuery": "",
        "Scheme": ""
      }
    }
  },
  "HostAddress": "xxxxxxxxx",
  "RequestKey": "78478050-47f0-4d0d-44e8-615d0599574a",
  "ServiceKey": "MAAS_SVC",
  "TimeUtc": "2016-09-23T14:13:02.217520245Z",
  "Title": "Get All Campaigns - Start",
  "beat": {
    "hostname": "host",
    "name": "host"
  },
  "input_type": "log",
  "offset": 919,
  "source": "input.json",
  "type": "log"
}

注意：您发布的 JSON 数据无效。 Cancel 字段缺少一个值。我在通过 Filebeat 运行数据之前将其设置为 null。

Answer 2

看起来像Kibana 7.2 (June 2019) does have RBAC now, with feature control

Want to hide Dev Tools from the left navigation? Show Stack Monitoring only to admins? Or, give certain users access to only Dashboard and Canvas? Feature controls allow you to hide and restrict applications and features in the Kibana UI.

You can configure Kibana applications and features based on your users’ needs, and when used with security, based on their privileges.

This means different roles can have access to different features in the same space. Power users might have privileges to create and edit visualizations and dashboards, while analysts or executives might have Dashboard and Canvas with read-only privileges.

使用filebeat将结构化日志数据直接推送到elasticsearch

Pushing structured log data directly to elastic search with filebeat

elasticsearch

elastic-stack

filebeat