弹性搜索批量 API、管道和地理 IP

Question

我使用批量 API 将数据导入我的 ELK 堆栈。

{"index":{"_index":"waf","_type":"logs","_id":"325d05bb6900440e"}}
{"id":"325d05bb6900440e","country":"US","ip":"1.1.1.1","protocol":"HTTP/1.1","method":"GET","host":"xxxxx","user_agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36","uri":"/?a=><script>alert(1)</script>","request_duration":1999872,"triggered_rule_ids":["100030"],"action":"challenge","cloudflare_location":"unknown","occurred_at":"2017-01-23T17:38:58.46Z","rule_detail":[{"id":"","description":"ARGS:A"}],"rule_message":"Generic XSS Probing","type":"waf","rule_id":"100030"}

我在数据中有一个 ip，我想使用 GEOIP 插件将其转换为经度和纬度。

我创建了一个管道：

PUT _ingest/pipeline/geoip-info
{
"description": "Add geoip info",
"processors": [
{
  "geoip": {
    "field": "ip",
    "target_field": "client_geoip",
    "properties": ["location"],
    "ignore_failure": true
  }
}
]
}`

然而，当我导入数据时，管道被忽略了。有人可以解释我如何修改批量 API 以通过管道传递信息，以便为我创建地图添加经度和纬度。

谢谢

Answer 1

在您的批量调用中，您缺少管道名称

                                                                    here
                                                                      |
                                                                      V
{"index":{"_index":"waf","_type":"logs","_id":"325d05bb6900440e", "pipeline": "geoip-info"}}
{"id":"325d05bb6900440e","country":"US","ip":"1.1.1.1","protocol":"HTTP/1.1","method":"GET","host":"xxxxx","user_agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36","uri":"/?a=><script>alert(1)</script>","request_duration":1999872,"triggered_rule_ids":["100030"],"action":"challenge","cloudflare_location":"unknown","occurred_at":"2017-01-23T17:38:58.46Z","rule_detail":[{"id":"","description":"ARGS:A"}],"rule_message":"Generic XSS Probing","type":"waf","rule_id":"100030"}

或者您也可以批量设置URL

POST _bulk?pipeline=geoip-info

弹性搜索批量 API、管道和地理 IP

Elastic Search Bulk API, Pipeline and Geo IP

elasticsearch

kibana

elasticsearch-bulk-api