EFK 系统构建于 docker 但 fluentd 无法启动

Question

我想通过 docker compose 构建 efk 记录器系统。一切都设置好了，只有 fluentd 有问题。

fluentd docker 容器日志

2022-02-15 02:06:11 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"

2022-02-15 02:06:11 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.3'

2022-02-15 02:06:11 +0000 [info]: gem 'fluentd' version '1.12.0'

/usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)

我的目录：

 my project/
 ├─ fluentd/
 │  ├─ conf/
 │  │  └── fluent.conf
 │  └── Dockerfile
 └── docker-compose.yml

docker-compose.yml:

version: "3"
services:
  web:
    image: httpd
    ports:
      - "8010:80"
    depends_on:
      - fluentd
    logging:
      driver: "fluentd"
      options:
        fluentd-address: 127.0.0.1:24224
        fluentd-async: 'true'
        tag: httpd.access

  fluentd:
    build: ./fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
    links:
      - "elasticsearch"
    ports:
      - "24224:24224"
      - "24224:24224/udp"

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.13.1
    environment:
      - discovery.type=single-node
    expose:
      - 9200
    ports:
      - "9200:9200"

  kibana:
    image: docker.elastic.co/kibana/kibana:7.13.1
    links:
      - "elasticsearch"
    ports:
      - "5601:5601"

Docker 文件：

# fluentd/Dockerfile

FROM fluent/fluentd:v1.12.0-debian-1.0
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.3"]
USER fluent

fluent.conf:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match *.**>
  @type copy

  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    flush_interval 1s
  </store>

  <store>
    @type stdout
  </store>
</match>

Answer 1

我遇到了同样的问题，但我过去常常制作完全相同的图像，直到今天一切正常。我不知道发生了什么变化。

但如果你需要紧急解决问题，请使用我的in-person图像：

docker pull kurraj/fluentd_castom:latest

我的 docker 文件：

FROM fluent/fluentd:v1.14-1

USER root

RUN gem update --system && \
gem install fluent-plugin-elasticsearch --source http://rubygems.org

Answer 2

好像elasticsearch 8.0 打破了这个功能。您可以等待修复，在等待时添加类似的内容：

FROM fluent/fluentd:v1.12.0-debian-1.0
USER root
RUN gem uninstall -I elasticsearch && gem install elasticsearch -v 7.17.0
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "-- 
 version", "5.0.3"]
USER fluent

Answer 3

我有同样的问题，我通过将 Dockerfile 中的 fluent-plugin-elasticsearch 版本更新为 5.2.0 来解决它。

EFK 系统构建于 docker 但 fluentd 无法启动

EFK system is build on docker but fluentd can't start up

elasticsearch

docker-compose

kibana

fluentd

efk