使用 Tshark 和 Flume 捕获网络流量

Question

大家好，我正在尝试使用 tshark 捕获网络流量，我正在使用 apache flume 将这些结果发送到 spark。

问题是当我在 flume 的配置 flume 中使用 exec 源时它不起作用它在启动后立即停止

我的配置：

agent.sources = tsharkSource 
agent.channels = memoryChannel 
agentasinks = avroSink 

# Configuring the sources to pull the bashes from Tshark 
agent.sources.tsharkSource.type = exec 
agent.sources.tsharkSource.command = tshark -T json 
agent.sources.tsharkSource.channels = memoryChannel 

# Configuring the sink to push logs to spark change hostname to 116's ip adress 
agent.sinks.avroSink.type = avro 
agent.sinks.avroSink.channel = memoryChannel 
agent.sinks.avroSink.hostname = 192.168.1.112 
agent.sinks.avroSink.port= 6969 

# Configuring the memory channel 
agent.channels.memoryChannel.type = memory 
agent.channels.memoryChannel.capacity = 1000 
agent.channels.memoryChannel.transactionCapacity = 100

shell 输出：

flume-ng agent -f conf/flume-conf.properties -n agent 

Warning: No configuration directory set! Use --conf <dir> to override. 
Info: Including Hive libraries found via () for Hive access 
+ exec /usr/lib/jum/jaua-1.11.0-openjdk-amd64//bin/jaua -Xmx20m -cp 
1/home/oshiflume/flume/apache-flume-1.8.0-bin/lib/*:/libfle - 
Djava.library.path= org.apache.flume.node.Application -f conf/flume- 
conf.properties -n agent 
log4j:WARN No appenders could be found for logger (org.apache.flume.node.Application). 
log4j:WARN Please initialize the log4j system properly. 
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.htmlDnoconfig for more info.

Answer 1

你的执行命令是

flume-ng agent -f conf/flume-conf.properties -n agent

我看到这里有两个错误。首先，必须指定配置目录-c conf，一般flume配置文件命名为some-config.conf

控制台的警告是No configuration directory set! Use --conf，-c和--conf是一回事

您可能希望将配置文件从 flume-conf.properties 重命名为 flume.conf

作为解决方案，您可以尝试此命令：

flume-ng agent -c conf -f conf/flume.conf -n agent

如果要在执行后显示日志，请使用此命令

flume-ng agent -c conf -f conf/flume.conf -n agent -Dflume.root.logger=INFO,console

要显示日志 log4j.properties 必须作为 conf/log4j.properties 在您的 conf 目录中。

我的属性如下：

flume.root.logger=INFO,LOGFILE
flume.log.dir=./logs
flume.log.file=flume.log

log4j.logger.org.apache.flume.lifecycle = INFO
log4j.logger.org.jboss = WARN
log4j.logger.org.mortbay = INFO
log4j.logger.org.apache.avro.ipc.NettyTransceiver = WARN
log4j.logger.org.apache.hadoop = INFO
log4j.logger.org.apache.hadoop.hive = ERROR

# Define the root logger to the system property "flume.root.logger".
log4j.rootLogger=${flume.root.logger}

log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.LOGFILE.MaxFileSize=100MB
log4j.appender.LOGFILE.MaxBackupIndex=10
log4j.appender.LOGFILE.File=${flume.log.dir}/${flume.log.file}
log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n

log4j.appender.DAILY=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.DAILY.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.DAILY.rollingPolicy.ActiveFileName=${flume.log.dir}/${flume.log.file}
log4j.appender.DAILY.rollingPolicy.FileNamePattern=${flume.log.dir}/${flume.log.file}.%d{yyyy-MM-dd}
log4j.appender.DAILY.layout=org.apache.log4j.PatternLayout
log4j.appender.DAILY.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n

使用 Tshark 和 Flume 捕获网络流量

Capturing networking traffic using Tshark and Flume

flume

flume-ng

tshark