如何使 fluentd 从上次动态创建的文件夹向 Loki 发送日志

Question

目前我收集了所有多行日志并在 Grafana 仪表板上将它们可视化。现在我需要从最后一个动态创建的文件夹中获取日志，并在单独的 Grafana 仪表板上将它们可视化。每次启动应用程序时都会动态创建文件夹。

日志路径是下一个：

/var/log/2021-01-13/12_57_12_201/
/var/log/2021-01-13/12_54_48_123/
/var/log/2021-01-14/11_15_23_849/11_15_23_849_A.log
/var/log/2021-01-14/11_15_23_849/11_15_23_849_B.log
/var/log/2021-01-14/11_15_23_849/11_15_23_849_C.log
/var/log/2021-01-14/11_15_23_849/11_15_23_849_D.log

我使用docker-compose文件启动grafana:7.3.3，fluent-plugin-loki:latest和loki:2.1.0，以及fluent.conf文件收集所有日志并将它们发送给 Loki。是否可以从上次动态创建的文件夹中收集日志？那怎么可能呢？

这是我的 fluent.conf 文件：

<source>
   @type tail
   @log_level debug
   path /var/log/%Y-%m-%d/*/*.log
   pos_file  /var/log/pos/positions.log.pos
   tag varlog.*
   path_key filename
   multiline_flush_interval 5s
   refresh_interval 1s
   read_from_head true
   follow_inodes true
   <parse>
      @type multiline
      format_firstline /^(?<time>\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{4})(?<message>.+)$/
      format1 /^(?<time>\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{4})(?<message>.+)$/
      time_key               time
      time_format %Y-%m-%d %H:%M:%S.%N
      timezone               +0200
    </parse>
</source>

<match varlog.**>
  @type loki
  url "http://loki:3100"
  <buffer>
    flush_interval 1s
    chunk_limit_size 1m
    flush_at_shutdown true
  </buffer>
  extra_labels {"job":"applogs", "agent":"fluentd"}
  <label>
    filename
  </label>
</match>

使用 fluentd exec 输入插件更改配置后，我在 fluentd 日志中收到以下信息： fluent/log.rb:302:debug: Executing command title=:exec_input spawn=[{}, "sh /var/log/folderParser.sh"] mode=[:read] stderr=:discard 每次应该启动 bash 脚本时都会出现。

我当前的 fluent.config 文件与 exec 输入插件具有下一个配置：

<source>
   @type exec
   command sh /var/log/folderParser.sh
   tag newlog.*
   run_interval 10s
   <parse>
      @type none
   </parse>
</source>

<source>
   @type tail
   @log_level debug
   path /var/log/lastLogs/*.log
   pos_file  /var/log/pos/positions.log.pos
   tag newlog.*
   path_key filename
   multiline_flush_interval 5s
   refresh_interval 1s
   read_from_head true
   <parse>
      @type multiline
      format_firstline /^(?<time>\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{4})(?<message>.+)$/
      format1 /^(?<time>\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{4})(?<message>.+)$/
      time_key               time
      time_format %Y-%m-%d %H:%M:%S.%N
      timezone               +0200
    </parse>
</source>

<match newlog.**>
  @type loki
  url "http://loki:3100"
  <buffer>
    flush_interval 1s
    chunk_limit_size 1m
    flush_at_shutdown true
  </buffer>
  extra_labels {"job":"applogs", "agent":"fluentd", "datacenter":"lastlogs"}
  <label>
    filename
  </label>
</match>

这里还有应该由exec插件执行的脚本（它的路径/var/log/）。所有日志都在这里 path /var/log/%Y-%m-%d/time_when_app_started/*.log

# Remove all existing files from the previous run
rm -r lastLogs

# Move to the needed folder - var
cd ../

# Find last created folder in log directory
date_folder=`ls -ltr ./log | grep '^d' | tail -1| awk '{print $NF}'`
cd log/
# Find last created folder in last date directory
last_created_folder=`ls -ltr ./$date_folder | grep '^d' | tail -1| awk '{print $NF}'`

mkdir lastLogs
#sleep 1m
cp ./$date_folder/$last_created_folder/* ./lastLogs

最新的bash脚本

pwd
cd /var/
pwd

# Find last created folder in log directory
date_folder=`ls -ltr ./log | grep '^d' | tail -1| awk '{print $NF}'`
echo $date_folder

# Change directory to log dir to be able to find the last created folder with the logs
cd log/
pwd

# Find last created folder in last date directory
last_created_folder=`ls -ltr ./$date_folder | grep '^d' | tail -1| awk '{print $NF}'`
echo $last_created_folder
# Find last created folder within time directories
path=$date_folder/$last_created_folder
echo $path

# Check if /lastLogs/logs directory contains last logs
if diff ./$date_folder/$last_created_folder/ ./lastLogs/logs/
then
    echo "Files are the same"
#    cp -u ./$date_folder/$last_created_folder/* ./lastLogs/logs

else
    echo "Files or files scope are different"
    rm -r ./lastLogs/logs/*
    cp -u ./$date_folder/$last_created_folder/* ./lastLogs/logs
    echo "Files are reload"
fi

这是我的 fluent.conf 文件

<source>
    @type exec
    command sh /var/log/folderParser.sh
    tag exec.*
    run_interval 1s
    <parse>
       @type none
    </parse>
</source>

<match exec.*>
  @type stdout
</match>

<source>
   @type tail
   @log_level debug
   path /var/log/lastLogs/logs/*.log
   pos_file  /var/log/pos/positions.log.pos
   tag newlog.*
   path_key filename
   multiline_flush_interval 5s
   refresh_interval 1s
   read_from_head true
   <parse>
      @type multiline
      format_firstline /^(?<time>\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{4})(?<message>.+)$/
      format1 /^(?<time>\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{4})(?<message>.+)$/
      time_key               time
      time_format %Y-%m-%d %H:%M:%S.%N
      timezone               +0200
    </parse>
</source>

<match newlog.**>
  @type loki
  url "http://loki:3100"
  <buffer>
    flush_interval 1s
    chunk_limit_size 1m
    flush_at_shutdown true
  </buffer>
  extra_labels {"job":"applogs", "agent":"fluentd", "datacenter":"lastlogs"}
  <label>
    filename
  </label>
</match>

Answer 1

一个可能的解决方案是：

将 exec 输入插件与 shell 脚本一起使用。
shell 脚本将评估最新文件并将其移动到预定义的 path/location。
您可以tail新文件的这个预定义路径。
此外，将 exec 的输出路由到 null。

如何使 fluentd 从上次动态创建的文件夹向 Loki 发送日志

How to make fluentd send logs to Loki from the last dynamically created folder

fluentd

grafana