Fluentd elasticsearch 插件未在 Raspberry Pi 上从 Kubernetes 连接到 Elasticsearch
Fluentd elasticsearch plugin not connecting to Elasticsearch from Kubernetes on a Raspberry Pi
编辑：我根据 efrat-levitan's 建议的评论添加了新信息。当我按照原始评论的建议将我的 Elasticsearch 版本升级到 7.6.0 时，列出的日志输出略有不同。为了帮助调试，我也没有立即启动 Elasticsearch。可以在日志中的 ECONNREFUSED 消息中看到此操作的效果。我在下面的摘要中指出了日志文件的变化。其余大部分消息文本（即，不是日志片段）与以前相同。
我一直在努力通过 docker 中的 fluent-plugin-elasticsearch 插件 运行 获得一个 ARM 版本（针对 Raspberry Pi 3 & 4）。我找不到合适的 docker 图片，所以我自己制作了一张（如果有人知道我在哪里可以找到一张，我将不胜感激）。我从 fluentd-docker-image repo (doesn't include Elasticsearch plugins) and modified it as I thought necessary using the fluentd-kubernetes-daemonset 存储库开始（确实包括 Elasticsearch 插件）。好消息是它在 Raspberry Pi 上启动得很好。坏消息是它似乎甚至没有尝试连接到 ElasticSearch（Raspberry Pi 网络外部。日志文件如下所示：
2020-03-02 18:13:15 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.12'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.4'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.3.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.6.1'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.1'
2020-03-02 18:13:16 +0000 [info]: gem 'fluentd' version '1.9.2'
2020-03-02 18:13:16 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-03-02 18:13:16 +0000 [info]: using configuration file: <ROOT>
<filter **>
@type stdout
</filter>
<source>
@type forward
@label @ES
</source>
<label @ES>
<match out.elasticsearch.**>
@type elasticsearch
@log_level "info"
include_tag_key true
host "10.0.0.223"
port 9200
path ""
scheme http
index_name "logstash"
include_timestamp true
log_es_400_reason false
logstash_prefix "logstash"
logstash_dateformat "%Y.%m.%d"
logstash_format true
ssl_verify true
ssl_version TLSv1_2
user
password xxxxxx
reload_connections false
reconnect_on_error true
reload_on_failure true
request_timeout 5s
sniffer_class_name "Fluent::Plugin::ElasticsearchSimpleSniffer"
type_name "doc"
template_name
template_file
template_overwrite false
time_key "@timestamp"
<buffer>
flush_thread_count 8
flush_interval 5s
chunk_limit_size 2M
queue_limit_length 32
retry_max_interval 30
retry_forever true
</buffer>
</match>
</label>
<label @ERROR>
<match **>
@type stdout
</match>
</label>
</ROOT>
2020-03-02 18:13:16 +0000 [info]: starting fluentd-1.9.2 pid=7 ruby="2.6.5"
2020-03-02 18:13:16 +0000 [info]: spawn command to main: cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "-r", "/usr/local/bundle/gems/fluent-plugin-elasticsearch-4.0.4/lib/fluent/plugin/elasticsearch_simple_sniffer.rb", "--under-supervisor"]
2020-03-02 18:13:20 +0000 [info]: adding match in @ES pattern="out.elasticsearch.**" type="elasticsearch"
2020-03-02 18:13:23 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:23 +0000 [warn]: #0 Remaining retry: 14. Retry to communicate after 2 second(s).
2020-03-02 18:13:27 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:27 +0000 [warn]: #0 Remaining retry: 13. Retry to communicate after 4 second(s).
2020-03-02 18:13:35 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:35 +0000 [warn]: #0 Remaining retry: 12. Retry to communicate after 8 second(s).
2020-03-02 18:13:51 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:51 +0000 [warn]: #0 Remaining retry: 11. Retry to communicate after 16 second(s).
2020-03-02 18:13:51 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2020-03-02 18:13:51 +0000 [info]: adding match in @ERROR pattern="**" type="stdout"
2020-03-02 18:13:51 +0000 [info]: adding filter pattern="**" type="stdout"
2020-03-02 18:13:51 +0000 [info]: adding source type="forward"
2020-03-02 18:13:51 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-03-02 18:13:51 +0000 [info]: #0 starting fluentd worker pid=22 ppid=7 worker=0
2020-03-02 18:13:51 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
2020-03-02 18:13:51 +0000 [info]: #0 fluentd worker is now running worker=0
2020-03-02 18:13:51.581170450 +0000 fluent.info: {"pid":22,"ppid":7,"worker":0,"message":"starting fluentd worker pid=22 ppid=7 worker=0"}
2020-03-02 18:13:51 +0000 [warn]: #0 no patterns matched tag="fluent.info"
2020-03-02 18:13:51.585716902 +0000 fluent.info: {"port":24224,"bind":"0.0.0.0","message":"listening port port=24224 bind=\"0.0.0.0\""}
2020-03-02 18:13:51 +0000 [warn]: #0 no patterns matched tag="fluent.info"
2020-03-02 18:13:51.593737828 +0000 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}
为了进一步缩小日志范围，fluentd 似乎在配置和连接方面都了解 Elasticsearch：
2020-03-02 18:13:20 +0000 [info]: adding match in @ES pattern="out.elasticsearch.**" type="elasticsearch"
2020-03-02 18:13:23 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
'ECONNREFUSED' 错误是因为我停止了 Elasticsearch 以帮助排除故障。所以 Fluentd 正在尝试连接。如下一个片段所示，启动 Elasticsearch 后，它确实连接并继续处理：
2020-03-02 18:13:51 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
问题是 Fluentd 似乎并没有真正完成 Elasticsearch 的 "sign" 过程。我希望在成功时看到类似这样的内容，或者某种错误消息。
2020-02-28 21:56:26 +0000 [info]: #0 [out_es] Connection opened to Elasticsearch cluster => {:host=>"10.0.0.223", :port=>9200, :scheme=>"http", :path=>""}
我还希望在 Elasticsearch 中看到它确实完成了 "signon" 过程的一些证据。例如，我在 Elasticsearch 中没有看到 "logstash" 索引，也没有在 Elasticsearch 日志中看到任何证据表明除 Kibana 之外的任何客户端都已连接。
上面记录的配置对我来说似乎是正确的。日志中记录的命令行对我来说也很好。
/usr/local/bin/ruby -Eascii-8bit:ascii-8bit /usr/local/bundle/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins -r /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.0.4/lib/fluent/plugin/elasticsearch_simple_sniffer.rb --under-supervisor
登录 pod 并连接到 Elasticsearch 也可以：
$ ks exec -it fluentd-h2qzn sh
$ curl http://10.0.0.223:9200
{
"name" : "Richs-MacBook.local",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "OkZ2-Lj2RjW-pVyVl0C7og",
"version" : {
"number" : "7.6.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "7f634e9f44834fbc12724506cc1da681b0c3b1e3",
"build_date" : "2020-02-06T00:09:00.449973Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
有人可以阐明缺乏与 Elasticsearch 主机的连接或提供一些我可以执行的额外故障排除步骤吗？
谢谢，
有钱人
如 Efrat Levitan 所述，问题确实是版本不匹配。我安装了 Elasticsearch 7.6.0 和相应的 Kibana 版本 7.6.0，它正在运行。 Fluentd 已启动并且 运行 和日志显示在 Kibana 中。
编辑：我根据 efrat-levitan's 建议的评论添加了新信息。当我按照原始评论的建议将我的 Elasticsearch 版本升级到 7.6.0 时，列出的日志输出略有不同。为了帮助调试，我也没有立即启动 Elasticsearch。可以在日志中的 ECONNREFUSED 消息中看到此操作的效果。我在下面的摘要中指出了日志文件的变化。其余大部分消息文本（即，不是日志片段）与以前相同。
我一直在努力通过 docker 中的 fluent-plugin-elasticsearch 插件 运行 获得一个 ARM 版本（针对 Raspberry Pi 3 & 4）。我找不到合适的 docker 图片，所以我自己制作了一张（如果有人知道我在哪里可以找到一张，我将不胜感激）。我从 fluentd-docker-image repo (doesn't include Elasticsearch plugins) and modified it as I thought necessary using the fluentd-kubernetes-daemonset 存储库开始（确实包括 Elasticsearch 插件）。好消息是它在 Raspberry Pi 上启动得很好。坏消息是它似乎甚至没有尝试连接到 ElasticSearch（Raspberry Pi 网络外部。日志文件如下所示：
2020-03-02 18:13:15 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.12'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.4'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.3.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.6.1'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.1'
2020-03-02 18:13:16 +0000 [info]: gem 'fluentd' version '1.9.2'
2020-03-02 18:13:16 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-03-02 18:13:16 +0000 [info]: using configuration file: <ROOT>
<filter **>
@type stdout
</filter>
<source>
@type forward
@label @ES
</source>
<label @ES>
<match out.elasticsearch.**>
@type elasticsearch
@log_level "info"
include_tag_key true
host "10.0.0.223"
port 9200
path ""
scheme http
index_name "logstash"
include_timestamp true
log_es_400_reason false
logstash_prefix "logstash"
logstash_dateformat "%Y.%m.%d"
logstash_format true
ssl_verify true
ssl_version TLSv1_2
user
password xxxxxx
reload_connections false
reconnect_on_error true
reload_on_failure true
request_timeout 5s
sniffer_class_name "Fluent::Plugin::ElasticsearchSimpleSniffer"
type_name "doc"
template_name
template_file
template_overwrite false
time_key "@timestamp"
<buffer>
flush_thread_count 8
flush_interval 5s
chunk_limit_size 2M
queue_limit_length 32
retry_max_interval 30
retry_forever true
</buffer>
</match>
</label>
<label @ERROR>
<match **>
@type stdout
</match>
</label>
</ROOT>
2020-03-02 18:13:16 +0000 [info]: starting fluentd-1.9.2 pid=7 ruby="2.6.5"
2020-03-02 18:13:16 +0000 [info]: spawn command to main: cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "-r", "/usr/local/bundle/gems/fluent-plugin-elasticsearch-4.0.4/lib/fluent/plugin/elasticsearch_simple_sniffer.rb", "--under-supervisor"]
2020-03-02 18:13:20 +0000 [info]: adding match in @ES pattern="out.elasticsearch.**" type="elasticsearch"
2020-03-02 18:13:23 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:23 +0000 [warn]: #0 Remaining retry: 14. Retry to communicate after 2 second(s).
2020-03-02 18:13:27 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:27 +0000 [warn]: #0 Remaining retry: 13. Retry to communicate after 4 second(s).
2020-03-02 18:13:35 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:35 +0000 [warn]: #0 Remaining retry: 12. Retry to communicate after 8 second(s).
2020-03-02 18:13:51 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:51 +0000 [warn]: #0 Remaining retry: 11. Retry to communicate after 16 second(s).
2020-03-02 18:13:51 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2020-03-02 18:13:51 +0000 [info]: adding match in @ERROR pattern="**" type="stdout"
2020-03-02 18:13:51 +0000 [info]: adding filter pattern="**" type="stdout"
2020-03-02 18:13:51 +0000 [info]: adding source type="forward"
2020-03-02 18:13:51 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-03-02 18:13:51 +0000 [info]: #0 starting fluentd worker pid=22 ppid=7 worker=0
2020-03-02 18:13:51 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
2020-03-02 18:13:51 +0000 [info]: #0 fluentd worker is now running worker=0
2020-03-02 18:13:51.581170450 +0000 fluent.info: {"pid":22,"ppid":7,"worker":0,"message":"starting fluentd worker pid=22 ppid=7 worker=0"}
2020-03-02 18:13:51 +0000 [warn]: #0 no patterns matched tag="fluent.info"
2020-03-02 18:13:51.585716902 +0000 fluent.info: {"port":24224,"bind":"0.0.0.0","message":"listening port port=24224 bind=\"0.0.0.0\""}
2020-03-02 18:13:51 +0000 [warn]: #0 no patterns matched tag="fluent.info"
2020-03-02 18:13:51.593737828 +0000 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}
为了进一步缩小日志范围，fluentd 似乎在配置和连接方面都了解 Elasticsearch：
2020-03-02 18:13:20 +0000 [info]: adding match in @ES pattern="out.elasticsearch.**" type="elasticsearch"
2020-03-02 18:13:23 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
'ECONNREFUSED' 错误是因为我停止了 Elasticsearch 以帮助排除故障。所以 Fluentd 正在尝试连接。如下一个片段所示，启动 Elasticsearch 后，它确实连接并继续处理：
2020-03-02 18:13:51 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
问题是 Fluentd 似乎并没有真正完成 Elasticsearch 的 "sign" 过程。我希望在成功时看到类似这样的内容，或者某种错误消息。
2020-02-28 21:56:26 +0000 [info]: #0 [out_es] Connection opened to Elasticsearch cluster => {:host=>"10.0.0.223", :port=>9200, :scheme=>"http", :path=>""}
我还希望在 Elasticsearch 中看到它确实完成了 "signon" 过程的一些证据。例如，我在 Elasticsearch 中没有看到 "logstash" 索引，也没有在 Elasticsearch 日志中看到任何证据表明除 Kibana 之外的任何客户端都已连接。
上面记录的配置对我来说似乎是正确的。日志中记录的命令行对我来说也很好。
/usr/local/bin/ruby -Eascii-8bit:ascii-8bit /usr/local/bundle/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins -r /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.0.4/lib/fluent/plugin/elasticsearch_simple_sniffer.rb --under-supervisor
登录 pod 并连接到 Elasticsearch 也可以：
$ ks exec -it fluentd-h2qzn sh
$ curl http://10.0.0.223:9200
{
"name" : "Richs-MacBook.local",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "OkZ2-Lj2RjW-pVyVl0C7og",
"version" : {
"number" : "7.6.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "7f634e9f44834fbc12724506cc1da681b0c3b1e3",
"build_date" : "2020-02-06T00:09:00.449973Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
有人可以阐明缺乏与 Elasticsearch 主机的连接或提供一些我可以执行的额外故障排除步骤吗？
谢谢， 有钱人
如 Efrat Levitan 所述，问题确实是版本不匹配。我安装了 Elasticsearch 7.6.0 和相应的 Kibana 版本 7.6.0，它正在运行。 Fluentd 已启动并且 运行 和日志显示在 Kibana 中。