通过 logstash 将数据从 mongodb 同步到 elasticsearch

Question

我只想使用 logstash 将数据从 mongodb 同步到 elastic。它运行良好，当 mongodb 中有任何新记录时，logstash 会推送到 elastic。但是当我更新 mongodb 中的任何记录时，即使我删除它也没有任何反应，它也不会变成 elasticsearch。我想在配置文件中进行更改，以便在 mongo 中更新或删除任何记录时，它也应该反映在 elastic 中。

input {
mongodb {

    uri => 'mongodb://xxxxxx:32769/database'
    placeholder_db_dir =>'/usr/share/logstash/bin/opt/logstash-mongodb/'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'tags'
}
}
filter {
mutate {
rename => { "_id" => "mongo_id" }
}
}
output {
    stdout {
            codec => rubydebug
    }
    elasticsearch {
            action => "index"
            index => "mongo_data"
            hosts => ["https://xxxxxxxx:8443"]
            ssl => true
  doc_as_upsert => true

   }
   }

Answer 1

这不是输入旨在支持的用例。 documentation states "This was designed for parsing logs that were written into mongodb. This means that it may not re-parse db entries that were changed and already parsed." For "may not" read "will not". The code builds a cursor that finds documents with an id greater than the last id it read. It never looks for updates or deletions. Note also that the test is "greater than the last id" and the way it initializes 最后一个 id 意味着它永远不会读取集合中的第一个文档。

通过 logstash 将数据从 mongodb 同步到 elasticsearch

sync data from mongodb to elasticsearch via logstash

mongodb

elasticsearch

logstash