如何在 flume 中设置日志文件名

Question

我正在使用 Apache flume 进行日志收集。这是我的配置文件

httpagent.sources = http-source
httpagent.sinks = local-file-sink
httpagent.channels = ch3

#Define source properties 

httpagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
httpagent.sources.http-source.channels = ch3
httpagent.sources.http-source.port = 8082


# Local File Sink

httpagent.sinks.local-file-sink.type = file_roll
httpagent.sinks.local-file-sink.channel = ch3
httpagent.sinks.local-file-sink.sink.directory = /home/avinash/log_dir
httpagent.sinks.local-file-sink.sink.rollInterval = 21600

# Channels

httpagent.channels.ch3.type = memory
httpagent.channels.ch3.capacity = 1000

我的应用程序正在运行 fine.My 问题是，在 log_dir 中，文件默认使用一些随机数（我猜是它的时间戳）时间戳。

如何为日志文件提供正确的文件名后缀？

Answer 1

查看 documentation it seems there is no parameter for configuring the name of the files that are going to be created. I've gone to the sources 寻找一些 hidden 参数，但没有 :)

进入实施细节，文件名似乎由 PathManager class:

管理

private PathManager pathController;
...
@Override
public Status process() throws EventDeliveryException {
    ...
    if (outputStream == null) {
        File currentFile = pathController.getCurrentFile();
        logger.debug("Opening output stream for file {}", currentFile);
        try {
            outputStream = new BufferedOutputStream(new FileOutputStream(currentFile));
    ...
}

正如您已经注意到的，它基于当前时间戳（显示构造函数和下一个文件 getter）：

public PathManager() {
    seriesTimestamp = System.currentTimeMillis();
    fileIndex = new AtomicInteger();
}

public File nextFile() {
    currentFile = new File(baseDirectory, seriesTimestamp + "-" + fileIndex.incrementAndGet());
    return currentFile;
}

因此，我认为您唯一的可能是扩展 File Roll 接收器并覆盖 process() 方法以使用自定义路径控制器。

Answer 2

对于来源，您已根据 shell 脚本执行命令来尾部和前置或附加详细信息。以下是示例：

# Describe/configure the source for tailing file
 httpagent.sources.source.type = exec
 httpagent.sources.source.shell = /bin/bash -c
 httpagent.sources.source.command = tail -F /path/logs/*_details.log
 httpagent.sources.source.restart = true
 httpagent.sources.source.restartThrottle = 1000
 httpagent.sources.source.logStdErr = true

如何在 flume 中设置日志文件名

How to set log filename in flume

apache

logging

flume

flume-ng

flume-twitter