Logstash

Question

大家好，我是 Elasticsearch 的新手，我有一个名为 'american_football_sacks_against_stats' 的 table。它由三列组成。

id	sacks_against_total	sacks_against_yards
1	12	5
2	15	3
...	...	...

问题是 sacks_against_total 和 sacks_against_yards 不是 'imported' 和 integers/longs/floats 一样，而是文本字段和关键字字段。如何将它们转换成数字？

我试过了，但没用：

mutate {
        convert => {
             "id" => "integer"
             "sacks_against_total" => "integer"
             "sacks_against_yards" => "integer"
        } 
    }

这是我的 logstash.conf 文件：

input {
    jdbc {
        jdbc_connection_string => "jdbc:postgresql://localhost:5432/sportsdb"
        jdbc_user => "user"
        jdbc_password => "password"
        jdbc_driver_class => "org.postgresql.Driver"
        schedule => "*/5 * * * *" 
        statement => "SELECT * FROM public.american_football_sacks_against_stats"
        jdbc_paging_enabled => "true"
        jdbc_page_size => "300"
    }
}

filter {
    mutate {
        convert => {
             "id" => "integer"
             "sacks_against_total" => "integer"
             "sacks_against_yards" => "integer"
        } 
    }
}

output {
    stdout { codec => "json" }
    elasticsearch {
        hosts => "http://localhost:9200"
        index => "sportsdb"
        doc_as_upsert => true #
    }
}

Answer 1

Elasticsearch 不提供更改现有字段类型的功能。

您可以在此处阅读一些选项： https://medium.com/@max.borysov/change-field-type-in-elasticsearch-index-2d11bb366517

Answer 2

这是我一直在寻找的解决方案： https://linuxhint.com/elasticsearch-reindex-change-field-type/

首先，您从索引创建输入并更改类型。

PUT _ingest/pipeline/convert_pipeline
{
  “description”: “converts the field sacks_against_total,sacks_against_yards fields to a long from string”,
  "processors" : [
    {
      "convert" : {
        "field" : "sacks_against_yards",
        "type": "long"
      }
    }, 
    {
      "convert" : {
        "field" : "sacks_against_total",
        "type": "long"
      }
    }
  ]
}

或在 cURL 中：

curl -XPUT "http://localhost:9200/_ingest/pipeline/convert_pipeline" -H 'Content-Type: application/json' -d'{  "description": "converts the sacks_against_total field to a long from string",  "processors" : [    {      "convert" : {        "field" : "sacks_against_total",        "type": "long"      }    }, {"convert" : {        "field" : "sacks_against_yards",        "type": "long"     } }  ]}'

然后将索引重新索引到另一个索引

POST _reindex
{
  “source”: {
    "index": "american_football_sacks_against_stats"
  },
  "dest": {
    "index": "american_football_sacks_against_stats_withLong",
    "pipeline": "convert_pipeline"
  }
}

或在 cURL 中：

curl -XPOST "http://localhost:9200/_reindex" -H 'Content-Type: application/json' -d'{  "source": {    "index": "sportsdb"  },  "dest": {    "index": "sportsdb_finish",    "pipeline": "convert_pipeline"  }}'

Logstash - 如何更改字段类型

Logstash - how to change field types

elasticsearch

kibana