向插件添加 ID 时 Logstash 6.2.4 崩溃（应为#之一）

Question

我正在尝试将 ID 字段添加到我的 Logstash 6.2.4 配置中。我想做的是调试“http://localhost:9600/_node/stats/pipelines" so i need some names (id is random UUIDs without id field in configs). I found documentation about plugin id。它对我来说是这样的：

input {
    http {
        port => "${HTTP_PORT_FOR_EVENTS:8089}"
        additional_codecs => {"application/json"=>"json"}
        id => "http_events"
        tags => [ "test" ]
    }
}

filter {
    if "test" in [tags]  {
        mutate {
            remove_field => [ "headers", "host" ]   
        }
    }
}

但是它崩溃是这样的：

input {
    http {
        port => "${HTTP_PORT_FOR_EVENTS:808}"
        additional_codecs => {"application/json"=>"json"}
        id => "http_events"
        tags => [ "test" ]
    }
}

filter {
    id => "test2"
    if "test" in [tags]  {
        mutate {
            remove_field => [ "headers", "host" ]   
        }
    }
}

有了这个错误（我猜，由于 Docker 容器重启或其他原因，有两个关闭的错误）：

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 32, column 8 (byte 676) after filter {\r\n id ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:105:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:ininterval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

SIGTERM received. Shutting down. Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} Ignoring the 'pipelines.yml' file because modules or command line options are specified Starting Logstash {"logstash.version"=>"6.2.4"} Successfully started Logstash API endpoint {:port=>9600} SIGTERM received. Shutting down.

Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} Ignoring the 'pipelines.yml' file because modules or command line options are specified Starting Logstash {"logstash.version"=>"6.2.4"} Successfully started Logstash API endpoint {:port=>9600}

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 24, column 8 (byte 534) after filter {\n id ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

这些配置也破坏了我的 logstash:

input {
    rabbitmq {
        id => "test1"
        type => "event"
        exchange => "event"
        exclusive => true
    }
}


input {
    id => "test1"
    rabbitmq {
        type => "event"
        exchange => "event"
        exclusive => true
    }
}

Answer 1

我找到了解决办法。

ID 必须放在过滤器插件字段（例如grok、mutate 或json），而不仅仅是过滤器字段：

filter {
    if "test" in [tags]  {
        mutate {
            id => "test2"
            remove_field => [ "headers", "host" ]   
        }
    }
}

也不是每个版本的插件都支持IP字段。必要时更新插件：

logstash-plugin update logstash-filter-mutate

向插件添加 ID 时 Logstash 6.2.4 崩溃（应为#之一）

Logstash 6.2.4 crashes when adding an ID to plugin (Expected one of #)

logstash

logstash-grok

logstash-configuration

elastic-stack