java.sql.SQLException：<field_name> 的验证失败：'null' 无法转换为双精度类型

Question

我正在使用 logstash 创建从 elasticsearch 到 crate.io 的管道。下面是配置。

input{
 elasticsearch {
   hosts => "<host_name>:9200"
   index => "index1"
   query => '{ "size":10,"query": {"match_all": {} } }'
   }
}
filter{
    if!([bench_pose][M_Body_t]) {
            mutate{
                    add_field => {"[bench_pose][M_Body_t]" => null}
            }
    }
    if!([bench_pose][M_Jaw_t]) {
            mutate{
                    add_field => {"[bench_pose][M_Jaw_t]" => null}
            }
    }
}
output{
 jdbc {
       driver_class => "io.crate.client.jdbc.CrateDriver"
       driver_auto_commit => false
       driver_jar_path => "/etc/crate/crate-jdbc-standalone-1.12.3.jar"
       connection_string => "crate://<host_ip>:4300"
       statement => ["INSERT INTO table_name(path,bench_pose_m_Body_t,bench_pose_m_jaw_t) VALUES(?,?,?)",'path','%{[bench_pose][M_Body_t]}','%{[bench_pose][M_Jaw_t]}']
   }
}

source有一个字段'bench_pose'是OBJECT数据类型，M_Body_t和M_Jaw_t下的字段bench_pose是DOUBLE类型。这些字段在源和目标中可以为空。该字段在源中的少数文档中具有空值。即使目标 table 具有 NULLable 字段，当插入语句遇到具有空值的文档时，也会抛出以下错误。

JDBC - Exception. Not retrying. {:exception=>java.sql.SQLException: Validation failed for bench_pose_m_body_t: 'null' cannot be cast to type double

如果我删除过滤器插件，它会抛出如下错误。

JDBC - Exception. Not retrying. {:exception=>java.sql.SQLException: Validation failed for bench_pose_m_body_t: '%{[bench_pose][M_Body_t]}' cannot be cast to type double

如果我在插入中硬编码空值并运行 crate sql 上的查询，它会正确执行。如何在 logstash 中解析这些字段中的空值？

Answer 1

您需要将 INSERT 语句更改为：

   statement => ["INSERT INTO table_name(path,benc‌h_pose_m_Body_t,bench‌_pose_m_jaw_t) VALUES(?,TRY_CAST(? as double),TRY_CAST(? as double))",'path','%{[bench_pose][M_Body_t]}','%{[bench_pose][M_Jaw_t]}']

TRY_CAST 将 return null 而不是在类型转换不兼容的情况下抛出错误。

java.sql.SQLException：<field_name> 的验证失败：'null' 无法转换为双精度类型

java.sql.SQLException: Validation failed for <field_name>: 'null' cannot be cast to type double

jdbc

elasticsearch

logstash

sqlexception

crate