如何在控制台上抑制 hadoop 生成的日志消息
how to suppress the logging messages generated by hadoop on the console
我正在使用 hadoop 2.5。2.I 是 hadoop 和 map reduce 的新手。我正在尝试使用 map reduce 代码来执行情绪 analysis.But 当我执行代码时，hadoop 在 console.Thats 上生成了很多日志消息，这是因为它需要花费大量时间来为每个消息提供分析结果sentence.Following 是日志消息....
2016-05-12 23:03:05,396 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-05-12 23:03:05,397 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-05-12 23:03:05,400 WARN mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-05-12 23:03:05,403 WARN mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(259)) - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-05-12 23:03:05,412 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 3
2016-05-12 23:03:05,421 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(396)) - number of splits:3
2016-05-12 23:03:05,430 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(479)) - Submitting tokens for job: job_local1427897879_0120
2016-05-12 23:03:05,444 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2016-05-12 23:03:05,445 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: hadoop.tmp.dir; Ignoring.
2016-05-12 23:03:05,445 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2016-05-12 23:03:05,482 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2016-05-12 23:03:05,482 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: hadoop.tmp.dir; Ignoring.
2016-05-12 23:03:05,483 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2016-05-12 23:03:05,483 INFO mapreduce.Job (Job.java:submit(1289)) - The url to track the job: http://localhost:8080/
2016-05-12 23:03:05,483 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1334)) - Running job: job_local1427897879_0120
2016-05-12 23:03:05,483 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2016-05-12 23:03:05,484 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2016-05-12 23:03:05,485 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2016-05-12 23:03:05,485 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1427897879_0120_m_000000_0
2016-05-12 23:03:05,486 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-12 23:03:05,486 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(462)) - Processing split: file:/home/hduser/workspace/Test1/training/pool.txt:0+17961
2016-05-12 23:03:05,487 INFO mapred.MapTask (MapTask.java:runOldMapper(416)) - numReduceTasks: 1
2016-05-12 23:03:05,487 INFO mapred.MapTask (MapTask.java:createSortingCollector(388)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-05-12 23:03:05,513 INFO mapred.MapTask (MapTask.java:setEquator(1182)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-12 23:03:05,513 INFO mapred.MapTask (MapTask.java:init(975)) - mapreduce.task.io.sort.mb: 100
2016-05-12 23:03:05,514 INFO mapred.MapTask (MapTask.java:init(976)) - soft limit at 83886080
2016-05-12 23:03:05,514 INFO mapred.MapTask (MapTask.java:init(977)) - bufstart = 0; bufvoid = 104857600
2016-05-12 23:03:05,514 INFO mapred.MapTask (MapTask.java:init(978)) - kvstart = 26214396; length = 6553600
2016-05-12 23:03:05,516 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2016-05-12 23:03:05,516 INFO mapred.MapTask (MapTask.java:flush(1437)) - Starting flush of map output
2016-05-12 23:03:05,516 INFO mapred.MapTask (MapTask.java:flush(1455)) - Spilling map output
2016-05-12 23:03:05,516 INFO mapred.MapTask (MapTask.java:flush(1456)) - bufstart = 0; bufend = 17961; bufvoid = 104857600
2016-05-12 23:03:05,516 INFO mapred.MapTask (MapTask.java:flush(1458)) - kvstart = 26214396(104857584); kvend = 26211024(104844096); length = 3373/6553600
2016-05-12 23:03:05,523 INFO mapred.MapTask (MapTask.java:sortAndSpill(1641)) - Finished spill 0
2016-05-12 23:03:05,524 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1427897879_0120_m_000000_0 is done. And is in the process of committing
2016-05-12 23:03:05,525 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/home/hduser/workspace/Test1/training/pool.txt:0+17961
2016-05-12 23:03:05,525 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1427897879_0120_m_000000_0' done.
2016-05-12 23:03:05,525 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1427897879_0120_m_000000_0
2016-05-12 23:03:05,525 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1427897879_0120_m_000001_0
2016-05-12 23:03:05,525 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-12 23:03:05,526 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(462)) - Processing split: file:/home/hduser/workspace/Test1/training/pool.txt~:0+17939
2016-05-12 23:03:05,526 INFO mapred.MapTask (MapTask.java:runOldMapper(416)) - numReduceTasks: 1
2016-05-12 23:03:05,527 INFO mapred.MapTask (MapTask.java:createSortingCollector(388)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:setEquator(1182)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:init(975)) - mapreduce.task.io.sort.mb: 100
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:init(976)) - soft limit at 83886080
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:init(977)) - bufstart = 0; bufvoid = 104857600
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:init(978)) - kvstart = 26214396; length = 6553600
2016-05-12 23:03:05,552 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
我已经提到了已经在 Whosebug [how to suppress Hadoop logging message on the console 上提出的类似问题，但它没有帮助我讨论明确的答案。我尝试使用某些论坛中建议的以下配置，但它不是 working.I 已尝试在 hadoop-env.sh
中进行设置
export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_ROOT_LOGGER="WARN,DRFA"
我还尝试通过设置以下值来编辑 log4j.properties 文件..
hadoop.root.logger=WARN,DRFA
hadoop.log.dir=.
hadoop.log.file=hadoop.log
但我仍然无法摆脱 hadoop mapreduce 在运行时生成的日志消息 execution.Its 延迟我在控制台上的输出，因为 well.Is 他们的任何解决方法或任何 java 代码，我可以将其嵌入到我的代码中并抑制 messages.any 建议，帮助..有人知道吗？
非常感谢！！
我找到了 this.All 的解决方案，它需要更改 mapreduce 的配置文件。
1.mapreduce.map.log.level 可以取值为 OFF、FATAL、ERROR、WARN、INFO、DEBUG、TRACE 和 ALL。如果设置了 "mapreduce.job.log4j-properties-file"，该设置可能会被覆盖。
- mapreduce.reduce.log.level 也可以取值 OFF、FATAL、ERROR、WARN、INFO、DEBUG、TRACE 和 ALL。如果 "mapreduce.job.log4j-properties-file" 是 set.So，则该设置可能会被覆盖，最好确保未设置 "mapreduce.job.log4j-properties-file"。
我们必须在 mapred-site.xml.
中设置以下属性
<property>
<name>mapreduce.map.log.level</name>
<value>OFF</value>
</property>
<property>
<name>mapreduce.reduce.log.level</name>
<value>OFF</value>
</property>
现在我在 console.But 上看不到任何日志消息它也有缺点，因为如果在执行 mapreduce 代码时发生任何错误，因为没有日志消息可见，我们无法找出任何错误。
我正在使用 hadoop 2.5。2.I 是 hadoop 和 map reduce 的新手。我正在尝试使用 map reduce 代码来执行情绪 analysis.But 当我执行代码时，hadoop 在 console.Thats 上生成了很多日志消息，这是因为它需要花费大量时间来为每个消息提供分析结果sentence.Following 是日志消息....
2016-05-12 23:03:05,396 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-05-12 23:03:05,397 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-05-12 23:03:05,400 WARN mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-05-12 23:03:05,403 WARN mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(259)) - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-05-12 23:03:05,412 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 3
2016-05-12 23:03:05,421 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(396)) - number of splits:3
2016-05-12 23:03:05,430 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(479)) - Submitting tokens for job: job_local1427897879_0120
2016-05-12 23:03:05,444 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2016-05-12 23:03:05,445 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: hadoop.tmp.dir; Ignoring.
2016-05-12 23:03:05,445 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2016-05-12 23:03:05,482 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2016-05-12 23:03:05,482 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: hadoop.tmp.dir; Ignoring.
2016-05-12 23:03:05,483 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2016-05-12 23:03:05,483 INFO mapreduce.Job (Job.java:submit(1289)) - The url to track the job: http://localhost:8080/
2016-05-12 23:03:05,483 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1334)) - Running job: job_local1427897879_0120
2016-05-12 23:03:05,483 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2016-05-12 23:03:05,484 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2016-05-12 23:03:05,485 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2016-05-12 23:03:05,485 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1427897879_0120_m_000000_0
2016-05-12 23:03:05,486 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-12 23:03:05,486 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(462)) - Processing split: file:/home/hduser/workspace/Test1/training/pool.txt:0+17961
2016-05-12 23:03:05,487 INFO mapred.MapTask (MapTask.java:runOldMapper(416)) - numReduceTasks: 1
2016-05-12 23:03:05,487 INFO mapred.MapTask (MapTask.java:createSortingCollector(388)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-05-12 23:03:05,513 INFO mapred.MapTask (MapTask.java:setEquator(1182)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-12 23:03:05,513 INFO mapred.MapTask (MapTask.java:init(975)) - mapreduce.task.io.sort.mb: 100
2016-05-12 23:03:05,514 INFO mapred.MapTask (MapTask.java:init(976)) - soft limit at 83886080
2016-05-12 23:03:05,514 INFO mapred.MapTask (MapTask.java:init(977)) - bufstart = 0; bufvoid = 104857600
2016-05-12 23:03:05,514 INFO mapred.MapTask (MapTask.java:init(978)) - kvstart = 26214396; length = 6553600
2016-05-12 23:03:05,516 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2016-05-12 23:03:05,516 INFO mapred.MapTask (MapTask.java:flush(1437)) - Starting flush of map output
2016-05-12 23:03:05,516 INFO mapred.MapTask (MapTask.java:flush(1455)) - Spilling map output
2016-05-12 23:03:05,516 INFO mapred.MapTask (MapTask.java:flush(1456)) - bufstart = 0; bufend = 17961; bufvoid = 104857600
2016-05-12 23:03:05,516 INFO mapred.MapTask (MapTask.java:flush(1458)) - kvstart = 26214396(104857584); kvend = 26211024(104844096); length = 3373/6553600
2016-05-12 23:03:05,523 INFO mapred.MapTask (MapTask.java:sortAndSpill(1641)) - Finished spill 0
2016-05-12 23:03:05,524 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1427897879_0120_m_000000_0 is done. And is in the process of committing
2016-05-12 23:03:05,525 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/home/hduser/workspace/Test1/training/pool.txt:0+17961
2016-05-12 23:03:05,525 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1427897879_0120_m_000000_0' done.
2016-05-12 23:03:05,525 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1427897879_0120_m_000000_0
2016-05-12 23:03:05,525 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1427897879_0120_m_000001_0
2016-05-12 23:03:05,525 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-12 23:03:05,526 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(462)) - Processing split: file:/home/hduser/workspace/Test1/training/pool.txt~:0+17939
2016-05-12 23:03:05,526 INFO mapred.MapTask (MapTask.java:runOldMapper(416)) - numReduceTasks: 1
2016-05-12 23:03:05,527 INFO mapred.MapTask (MapTask.java:createSortingCollector(388)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:setEquator(1182)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:init(975)) - mapreduce.task.io.sort.mb: 100
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:init(976)) - soft limit at 83886080
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:init(977)) - bufstart = 0; bufvoid = 104857600
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:init(978)) - kvstart = 26214396; length = 6553600
2016-05-12 23:03:05,552 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
我已经提到了已经在 Whosebug [how to suppress Hadoop logging message on the console 上提出的类似问题，但它没有帮助我讨论明确的答案。我尝试使用某些论坛中建议的以下配置，但它不是 working.I 已尝试在 hadoop-env.sh
中进行设置export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_ROOT_LOGGER="WARN,DRFA"
我还尝试通过设置以下值来编辑 log4j.properties 文件..
hadoop.root.logger=WARN,DRFA
hadoop.log.dir=.
hadoop.log.file=hadoop.log
但我仍然无法摆脱 hadoop mapreduce 在运行时生成的日志消息 execution.Its 延迟我在控制台上的输出，因为 well.Is 他们的任何解决方法或任何 java 代码，我可以将其嵌入到我的代码中并抑制 messages.any 建议，帮助..有人知道吗？
非常感谢！！
我找到了 this.All 的解决方案，它需要更改 mapreduce 的配置文件。
1.mapreduce.map.log.level 可以取值为 OFF、FATAL、ERROR、WARN、INFO、DEBUG、TRACE 和 ALL。如果设置了 "mapreduce.job.log4j-properties-file"，该设置可能会被覆盖。
- mapreduce.reduce.log.level 也可以取值 OFF、FATAL、ERROR、WARN、INFO、DEBUG、TRACE 和 ALL。如果 "mapreduce.job.log4j-properties-file" 是 set.So，则该设置可能会被覆盖，最好确保未设置 "mapreduce.job.log4j-properties-file"。
我们必须在 mapred-site.xml.
中设置以下属性<property>
<name>mapreduce.map.log.level</name>
<value>OFF</value>
</property>
<property>
<name>mapreduce.reduce.log.level</name>
<value>OFF</value>
</property>
现在我在 console.But 上看不到任何日志消息它也有缺点，因为如果在执行 mapreduce 代码时发生任何错误，因为没有日志消息可见，我们无法找出任何错误。