使用 logstash jdbc 将数组数据 mysql 加载到 ElasticSearch

Question

嗨，我是 ES 的新手，我正在尝试将数据从 'MYSQL' 加载到 'Elasticsearch'

我在尝试加载数组格式的数据时出现以下错误，求助

这是 mysql 数据，需要新的十六进制值列的数组数据

cid  color      new     hex      create            modified
1    100 euro   abcd   #86c67c  5/5/2016 15:48   5/13/2016 14:15
1    100 euro    1234   #fdf8ff  5/5/2016 15:48   5/13/2016 14:15

这是我们的 logstash 配置

input {
 jdbc {
   jdbc_driver_library => "/etc/logstash/mysql/mysql-connector-java-5.1.39-bin.jar"
   jdbc_driver_class => "com.mysql.jdbc.Driver"
   jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/test"
   jdbc_user => "root"
   jdbc_password => "*****"
   schedule => "* * * * *"

   statement => "select cid,color, new as 'cvalue.new',hexa_value as 'cvalue.hexa',created,modified from colors_hex_test order by cid"
   jdbc_paging_enabled => "true"
   jdbc_page_size => "50000"
}
}

output {
    elasticsearch {
        index => "colors_hexa"
        document_type => "colors"
        document_id => "%{cid}"
        hosts => "localhost:9200"

需要 cvalue（新的，hexa）的数组数据，如

{
  "_index": "colors_hexa",
  "_type": "colors",
  "_id": "1",
  "_version": 218,
  "found": true,
  "_source": {
    "cid": 1,
    "color": "100 euro",
    "cvalue" : {
            "new": "1234",
            "hexa_value": "#fdf8ff",
        }
    "created": "2016-05-05T10:18:51.000Z",
    "modified": "2016-05-13T08:45:30.000Z",
    "@version": "1",
    "@timestamp": "2016-05-14T01:30:00.059Z"
  }
}

这是我在运行 logstash

时遇到的错误

 "status"=>400, "error"=>{"type"=>"mapper_parsing_exception",
 "reason"=>"Field name [cvalue.hexa] cannot contain '.'"}}}, :level=>:warn}

Answer 1

您不能使用 . 给出字段名称。但是你可以尝试添加：

filter {
  mutate {
    rename => { "new" => "[cvalue][new]" }
    rename => { "hexa" => "[cvalue][hexa]" }
  }
}

使用 logstash jdbc 将数组数据 mysql 加载到 ElasticSearch

load array data mysql to ElasticSearch using logstash jdbc

elasticsearch

logstash

kibana-4

logstash-configuration