在 elasticsearch 和 Kibana 中导入 .log 文件

Question

我有这个带有 JSON 的 .log 文件，它看起来像这样

    {"method":"GET","path":"/public/index","format":"html","controller":"public","action":"index","status":200,"duration":84.59,"view":33.7,"db":47.45,"ip":"127.0.0.1","route":"public#index","request_id":"4d7016832294bafa8f593453eed2adb1","source":"unknown","tags":["request"],"@timestamp":"2018-11-09T22:54:06Z","@version":"1"}
{"method":"GET","path":"/public/index","format":"html","controller":"public","action":"index","status":200,"duration":15.44,"view":13.85,"db":0.91,"ip":null,"route":null,"request_id":null,"source":"unknown","tags":["request"],"@timestamp":"2018-11-09T22:54:28Z","@version":"1"}
{"method":"GET","path":"/public/index","format":"html","controller":"public","action":"index","status":200,"duration":13.86,"view":12.47,"db":0.8,"ip":null,"route":null,"request_id":null,"source":"unknown","tags":["request"],"@timestamp":"2018-11-09T22:54:40Z","@version":"1"}

我尝试在 elasticsearch 和 Kibana 中导入它。我在命令行中尝试使用此代码

curl -XPOST "http://localhost:9200/test/test" -H "Content-Type: application/json" -d @logfile.log

我收到这个错误

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Malformed content, found extra data after parsing: START_OBJECT"}},"status":400}

当我查看 Kibana 时，我看到了这个

我做错了什么？

Answer 1

批量格式为

action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....

您的文件中已经包含所有 optional_source 行。

只需添加一行包含 { "index" : { "_index" : "YOUR-INDEX-NAME", "_type" : "_doc"} } 在 each 行之前。

然后 POST 对抗 ES https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html

中的散装 api

在 elasticsearch 和 Kibana 中导入 .log 文件

import .log file in elasticsearch and Kibana

json

elasticsearch

curl

kibana

logfile