Flume 内存通道在启动时已满

Question

我的 Flume 内存通道有问题。我运行一个 Flume 代理，它淹没了内存通道并且日志开始溢出 "The channel is full, and cannot write data now. The source will try again after 250 milliseconds"

到目前为止一切顺利。我停止代理，编辑 flume.conf 以增加容量并重试。问题是在启动时 Flume 已经溢出相同的消息：

`16/05/14 00:21:48 INFO node.Application: Starting new configuration:
 { sourceRunners:{s1=EventDrivenSourceRunner: { source:Spool  
  Directory source s1: { spoolDir: /home/vagrant/logs } }} 
  sinkRunners:{kafka-avro-sink2=SinkRunner: { 
  policy:org.apache.flume.sink.DefaultSinkProcessor@63203b59 
  counterGroup:{ name:null counters:{} } }, kafka-avro-
   sink1=SinkRunner: { 
   policy:org.apache.flume.sink.DefaultSinkProcessor@591882e6 
  counterGroup:{ name:null counters:{} } }} channels:
  {mem1=org.apache.flume.channel.MemoryChannel{name: mem1}} }
  16/05/14 00:21:48 INFO node.Application: Starting Channel mem1
   16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup:       
  Monitored counter group for type: CHANNEL, name: mem1: Successfully  
  registered new MBean.
   16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup: 
  Component type: CHANNEL, name: mem1 started
  16/05/14 00:21:48 INFO node.Application: Starting Sink kafka-avro- 
  sink2
  16/05/14 00:21:48 INFO node.Application: Starting Sink kafka-avro-sink1
  16/05/14 00:21:48 INFO node.Application: Starting Source s1
  16/05/14 00:21:48 INFO source.SpoolDirectorySource:  
  SpoolDirectorySource source starting with directory:  
  /home/vagrant/logs
  16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup: 
  Monitored counter group for type: SOURCE, name: s1: Successfully  
  registered new MBean.
  16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup: 
   Component type: SOURCE, name: s1 started
  16/05/14 00:21:49 WARN source.SpoolDirectorySource: The channel is  
  full, and cannot write data now. The source will try again after 
  250 milliseconds
  16/05/14 00:21:49 INFO avro.ReliableSpoolingFileEventReader: Last  
  read was never committed - resetting mark position.
  16/05/14 00:21:49 WARN source.SpoolDirectorySource: The channel is  
  full, and cannot write data now. The source will try again after  
  500 milliseconds`

所以当我启动代理时频道已经满了。如何手动重置或清除它？

我已经为类似问题搜索了几个小时，但没有成功。一个有点令人恼火的问题是 https://flume.apache.org/ is so full of text that it basically gets indexed by google for pretty much anything flume related. So a search on "flume channel is always full" for example will invariably give https://flume.apache.org/ 作为第一结果。

Answer 1

您似乎无法为您的频道分配足够的内存。其中，您的事件传入率高于内存通道可用的内存。

尝试使用如下示例： xmx 是示例选项，您可以逐渐增加并根据您的代理数量运行查看内存分配的最佳值。

例如：

flume-ng agent -n $agentnumber -c ../../config/conf/ -f ../../config/conf/youragentconf.conf -Xmx3g

此外，您还需要查看代理配置文件中的其他配置参数，参考this Memory channel from Flume Guide。

Flume 内存通道在启动时已满

Flume memory channel is full on start-up

memory

hadoop

channel

flume