Logstash 在连接 AWS Elasticservice 时抛出 401
Logstash throwing 401 while connecting with AWS Elasticservice
我有 AWS Elastic 服务域设置,我正在尝试通过 Logstash 将一些数据从 ec2 实例推送到 AWS elasticservice。
我已经使用 Open access policy - Allow all traffic
设置了 AWS ES 域。我还启用了 Fine Grained Control
并设置了一个 master user
帐户来访问 AWS ES 服务。
使用 Kibana
或常规 cURL
调用一切正常,但 logstash 失败,请求发送到 https://<my_es_hostname>/_license
并返回 401
。
我不明白为什么会发生此呼叫。当我尝试在浏览器中点击它时,我得到 {"Message":"Your request: '/_license' is not allowed."}
这是我从 logstash 获得的示例日志:
[INFO ] 2021-06-02 11:40:18.858 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://<host-name-partxxxx>.us-east-2.es.amazonaws.com:443"]}
[INFO ] 2021-06-02 11:40:19.902 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/]}}
[WARN ] 2021-06-02 11:40:20.760 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/"}
[INFO ] 2021-06-02 11:40:21.371 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.10.2) {:es_version=>7}
[WARN ] 2021-06-02 11:40:21.380 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[ERROR] 2021-06-02 11:40:21.443 [[main]-pipeline-manager] elasticsearch - Unable to get license information {:url=>"https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/_license'"}
[ERROR] 2021-06-02 11:40:21.449 [[main]-pipeline-manager] elasticsearch - Could not connect to a compatible version of Elasticsearch {:url=>"https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/"}
这是我的 logstash 配置:
input {
jdbc {
jdbc_driver_class => "org.postgresql.Driver"
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.2.20.jar"
...
<other properties to fetch data>
}
}
output {
elasticsearch {
hosts => ["https://<host-name-partxxxx>.us-east-2.es.amazonaws.com:443"]
user => "username"
password => "password"
ilm_enabled => false
index => "my_index"
document_id => "%{id}"
doc_as_upsert => true
}
}
问题出在 Logstash Elasticsearch 输出插件试图验证 URL <hostname>/_license
.
上的许可证
参考 LS should always perform ES license check · Issue #1004 · logstash-plugins/logstash-output-elasticsearch 报告 issue/fix。
在该修复程序发布后,您可以按照以下步骤使其正常运行:
- 前往
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch
- 打开文件
license_checker.rb
-> 根据上述 github 问题中建议的修复方法更改方法 appropriate_license
。在 OSS
设置的情况下,使方法 appropriate_license()
return 为真。
logstash 有一个 Opensearch plugin 可以解决这个问题。
cd /usr/share/logstash
bin/logstash-plugin install logstash-output-opensearch
更新 Logstash 输出插件,将 elasticsearch
替换为 opensearch
output {
opensearch {
hosts => "<Your ES Host>"
... <OTHER CONFIG>
}
}
我有 AWS Elastic 服务域设置,我正在尝试通过 Logstash 将一些数据从 ec2 实例推送到 AWS elasticservice。
我已经使用 Open access policy - Allow all traffic
设置了 AWS ES 域。我还启用了 Fine Grained Control
并设置了一个 master user
帐户来访问 AWS ES 服务。
使用 Kibana
或常规 cURL
调用一切正常,但 logstash 失败,请求发送到 https://<my_es_hostname>/_license
并返回 401
。
我不明白为什么会发生此呼叫。当我尝试在浏览器中点击它时,我得到 {"Message":"Your request: '/_license' is not allowed."}
这是我从 logstash 获得的示例日志:
[INFO ] 2021-06-02 11:40:18.858 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://<host-name-partxxxx>.us-east-2.es.amazonaws.com:443"]}
[INFO ] 2021-06-02 11:40:19.902 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/]}}
[WARN ] 2021-06-02 11:40:20.760 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/"}
[INFO ] 2021-06-02 11:40:21.371 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.10.2) {:es_version=>7}
[WARN ] 2021-06-02 11:40:21.380 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[ERROR] 2021-06-02 11:40:21.443 [[main]-pipeline-manager] elasticsearch - Unable to get license information {:url=>"https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/_license'"}
[ERROR] 2021-06-02 11:40:21.449 [[main]-pipeline-manager] elasticsearch - Could not connect to a compatible version of Elasticsearch {:url=>"https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/"}
这是我的 logstash 配置:
input {
jdbc {
jdbc_driver_class => "org.postgresql.Driver"
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.2.20.jar"
...
<other properties to fetch data>
}
}
output {
elasticsearch {
hosts => ["https://<host-name-partxxxx>.us-east-2.es.amazonaws.com:443"]
user => "username"
password => "password"
ilm_enabled => false
index => "my_index"
document_id => "%{id}"
doc_as_upsert => true
}
}
问题出在 Logstash Elasticsearch 输出插件试图验证 URL <hostname>/_license
.
参考 LS should always perform ES license check · Issue #1004 · logstash-plugins/logstash-output-elasticsearch 报告 issue/fix。
在该修复程序发布后,您可以按照以下步骤使其正常运行:
- 前往
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch
- 打开文件
license_checker.rb
-> 根据上述 github 问题中建议的修复方法更改方法appropriate_license
。在OSS
设置的情况下,使方法appropriate_license()
return 为真。
logstash 有一个 Opensearch plugin 可以解决这个问题。
cd /usr/share/logstash
bin/logstash-plugin install logstash-output-opensearch
更新 Logstash 输出插件,将 elasticsearch
替换为 opensearch
output {
opensearch {
hosts => "<Your ES Host>"
... <OTHER CONFIG>
}
}