logstash 的 JDBC 输入插件可以执行多个 sql 任务吗？

Question

现在有一个 JDBC 任务从数据库读取为

input {
  jdbc {
    jdbc_connection_string => "jdbc:mysql://${MYSQL_MAIN_HOST}/${MYSQL_DATABASE}"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    jdbc_page_size => 10000
    jdbc_paging_enabled => true
    jdbc_password => "${MYSQL_PASSWORD}"
    jdbc_user => "${MYSQL_USER}"
    schedule => "0 1 * * *"
    statement_filepath => "/usr/share/logstash/pipeline/sql/select_posts.sql"
    tracking_column => "updated_at"
    tracking_column_type => "numeric"
    use_column_value => true
    last_run_metadata_path => "/usr/share/logstash/jdbc_last_run/select_posts_last_value"
  }
}

/usr/share/logstash/pipeline/sql/select_posts.sql

SELECT
  id,
  title,
  body
FROM posts

当 body 项目涉及非常大的数据时，此任务很繁重。所以我想在第一次搜索时将其删除为：

SELECT
  id,
  title
FROM posts

然后获取 ID 并使用它们再次查找 body。

SELECT
  body
FROM posts
WHERE id in (IDs)

设置为输出目标。甚至更新输出目标也可以。

那么在这种情况下logstash可以逐条读取吗？

Answer 1

如果您想使用 logstash 执行两个单独的 sql 查询，您可以按如下方式进行

查询1。 sql

SELECT
  id,
  title
FROM posts

query2.sql

SELECT
  body
FROM posts
WHERE id in (IDs)

logstash.conf

input {
  jdbc {
    jdbc_connection_string => "jdbc:mysql://${MYSQL_MAIN_HOST}/${MYSQL_DATABASE}"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    jdbc_page_size => 10000
    jdbc_paging_enabled => true
    jdbc_password => "${MYSQL_PASSWORD}"
    jdbc_user => "${MYSQL_USER}"
    schedule => "0 1 * * *"
    statement_filepath => <PATH TO query1.sql>  -----query1.sql
    tracking_column => "updated_at"
    tracking_column_type => "numeric"
    use_column_value => true
    last_run_metadata_path => "/usr/share/logstash/jdbc_last_run/select_posts_last_value"
  }

 jdbc {
    jdbc_connection_string => "jdbc:mysql://${MYSQL_MAIN_HOST}/${MYSQL_DATABASE}"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    jdbc_page_size => 10000
    jdbc_paging_enabled => true
    jdbc_password => "${MYSQL_PASSWORD}"
    jdbc_user => "${MYSQL_USER}"
    schedule => "0 1 * * *"
    statement_filepath => "<PATH TO query2.sql>"    -----query2.sql
    tracking_column => "updated_at"
    tracking_column_type => "numeric"
    use_column_value => true
    last_run_metadata_path => "/usr/share/logstash/jdbc_last_run/select_posts_last_value"
  }
}

logstash 的 JDBC 输入插件可以执行多个 sql 任务吗？

Can logstash's JDBC input plugin do multiple sql tasks?

mysql

sql

jdbc

logstash

logstash-jdbc