Apache Flume spoolDirectory 配置失败

Question

我正在使用以下代码将源目录中的文件写入 hdfs。

  # Initialize agent's source, channel and sink
agent.sources = test
agent.channels = memoryChannel
agent.sinks = flumeHDFS

# Setting the source to spool directory where the file exists
agent.sources.test.type = spooldir
agent.sources.test.spoolDir = /Data

# Setting the channel to memory
agent.channels.memoryChannel.type = memory
# Max number of events stored in the memory channel
agent.channels.memoryChannel.capacity = 10000
# agent.channels.memoryChannel.batchSize = 15000
agent.channels.memoryChannel.transactioncapacity = 1000000

# Setting the sink to HDFS
agent.sinks.flumeHDFS.type = hdfs
agent.sinks.flumeHDFS.hdfs.path = /user/team
agent.sinks.flumeHDFS.hdfs.fileType = DataStream

# Write format can be text or writable
agent.sinks.flumeHDFS.hdfs.writeFormat = Text

# use a single csv file at a time
agent.sinks.flumeHDFS.hdfs.maxOpenFiles = 1

# rollover file based on maximum size of 10 MB
agent.sinks.flumeHDFS.hdfs.rollCount=0
agent.sinks.flumeHDFS.hdfs.rollInterval=2000
agent.sinks.flumeHDFS.hdfs.rollSize = 0
agent.sinks.flumeHDFS.hdfs.batchSize =1000000

# never rollover based on the number of events
agent.sinks.flumeHDFS.hdfs.rollCount = 0

# rollover file based on max time of 1 min
#agent.sinks.flumeHDFS.hdfs.rollInterval = 0
# agent.sinks.flumeHDFS.hdfs.idleTimeout = 600

# Connect source and sink with channel
agent.sources.TwitterExampleDir.channels = memoryChannel
agent.sinks.flumeHDFS.channel = memoryChannel

但是我收到以下错误

: Failed to configure component! org.apache.flume.conf.ConfigurationException: Failed to configure component! at org.apache.flume.conf.source.SourceConfiguration.configure(SourceConfigurati on.java:110) at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSources( FlumeConfiguration.java:566) at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeCon figuration.java:345) at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.access[=12=]0(Flume Configuration.java:212) at org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfigur ation.java:126) at org.apache.flume.conf.FlumeConfiguration.(FlumeConfiguration.java:108) at org.apache.flume.node.PropertiesFileConfigurationProvider.getFlumeConfigurat ion(PropertiesFileConfigurationProvider.java:193) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(Abstrac tConfigurationProvider.java:94) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcher Runnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$ 301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch eduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11 42) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6 17) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.flume.conf.ConfigurationException: No channels set for test at org.apache.flume.conf.source.SourceConfiguration.configure(SourceConfigurati on.java:68) ... 15 more

谁能帮我把我的数据从源目录/Data 放到hdfs 目录/user/team？

Answer 1

堆栈跟踪提及：

No channels set for test

您已将 sources 指定为 test

agent.sources = test

但是在连接到 channel

时

agent.sources.TwitterExampleDir.channels = memoryChannel

所以你必须提到 test 而不是 TwitterExampleDir

Apache Flume spoolDirectory 配置失败

Apache Flume spoolDirectory configuration is failing

hadoop

flume

flume-ng