如果我不定义 start_position，Logstash 不会解析日志

Question

下面是我的代码。当我尝试运行这条管道时。管道正在启动但未解析日志。我已经删除了位于 (data/plugins/inputs/file).

的 .sincedb 文件

input {
    file {
        path => ".../directory/*.log"
    }
}
filter {
    
}
output { 
    stdout{
        codec => rubydebug
    }
}

但是如果我放，

start_position => "beginning"

行得通。但每次它都从头开始，而不是从它离开的地方开始。如果我输入以下代码并删除 .sincedb 文件。它仍然没有解析。

start_position => "end"

在第一种情况下，我没有定义 start_position 或定义 start_position =>“结束”。等待在“成功启动 Logstash API 端点 {:port=>9601}” 但是这两种情况我都删除了 .sincedb 文件。我怎样才能让 logstash 在它离开的地方继续？感谢您的回答

Answer 1

如文件输入插件文档中所述，start_position 仅在第一次读取文件时被考虑。

所以应该可以。使用 start_position => "beginning" 它应该第一次从开头开始，然后从 .sincedb 文件中写入的最后位置继续。

This option only modifies "first contact" situations where a file is new and not seen before, i.e. files that don’t have a current position recorded in a sincedb file read by Logstash. If a file has already been seen before, this option has no effect and the position recorded in the sincedb file will be used.

确保logstash 具有对数据目录的写入权限。也尝试减少 sincedb_write_interval.

最后，如文档中所述，sincedb 记录可能会过期。如果您尝试读取许多文件，最好使用 filebeat 读取日志并将它们传递给 logstash，而不是使用 logstash 的文件输入插件和 sincedb。

Sincedb records can now be expired meaning that read positions of older files will not be remembered after a certain time period. File systems may need to reuse inodes for new content. Ideally, we would not use the read position of old content, but we have no reliable way to detect that inode reuse has occurred. This is more relevant to Read mode where a great many files are tracked in the sincedb. Bear in mind though, if a record has expired, a previously seen file will be read again

如果我不定义 start_position，Logstash 不会解析日志

Logstash doesn't parse logs if i don't define start_position

logstash