在 kibana 中创建索引删除 .raw 字段时的 Logstash 问题

Question

我写了一个用于读取日志的 logstash conf 文件。如果我使用默认索引，即 logstash-*，我可以在 kibana 中看到 .raw 字段。但是，如果我在 logstash 的 conf 文件中创建一个新索引，如

output{
    elasticsearch {
      hosts => "localhost"
      index => "batchjob-*"} 
}

那么新索引无法配置.raw字段。有什么解决办法吗？非常感谢。

Answer 1

原始字段由 specific index template 创建，Logstash elasticsearch 输出在 Elasticsearch 中创建。

您只需将该模板复制到名为 batchjob.json 的文件中，然后将模板名称更改为 batchjob-*（见下文）

{
  "template" : "batchjob-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
      "_all" : {"enabled" : true, "omit_norms" : true},
      "dynamic_templates" : [ {
        "message_field" : {
          "match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "string", "index" : "analyzed", "omit_norms" : true,
            "fielddata" : { "format" : "disabled" }
          }
        }
      }, {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "string", "index" : "analyzed", "omit_norms" : true,
            "fielddata" : { "format" : "disabled" },
            "fields" : {
              "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
            }
          }
        }
      } ],
      "properties" : {
        "@timestamp": { "type": "date" },
        "@version": { "type": "string", "index": "not_analyzed" },
        "geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "float" },
            "longitude" : { "type" : "float" }
          }
        }
      }
    }
  }
}

然后您可以像这样修改 elasticsearch 输出：

output {
    elasticsearch {
      hosts => "localhost"
      index => "batchjob-*"
      template_name => "batchjob"
      template => "/path/to/batchjob.json"
    } 
}

在 kibana 中创建索引删除 .raw 字段时的 Logstash 问题

Logstash issues in creating index remove .raw field in kibana

elasticsearch

logstash

kibana