找到接口 org.apache.hadoop.mapreduce.TaskAttemptContext
Found interface org.apache.hadoop.mapreduce.TaskAttemptContext
到目前为止还没有找到解决我的特定问题的方法。它至少不起作用。这让我很疯狂。这个特殊的组合在 google space 中似乎没有太多。据我所知，我的错误发生在作业进入映射器时。该作业的输入是 avro 模式的输出，虽然我也尝试过未压缩，但它是用 deflate 压缩的。
阿芙罗：1.7.7
Hadoop：2.4.1
我遇到了这个错误，我不确定为什么。这是我的工作，mapper 和 reduce。映射器进入时发生错误。
示例未压缩的 Avro 输入文件（StockReport.SCHEMA 是这样定义的）
{"day": 3, "month": 2, "year": 1986, "stocks": [{"symbol": "AAME", "timestamp": 507833213000, "dividend": 10.59}]}
工作
@Override
public int run(String[] strings) throws Exception {
Job job = Job.getInstance();
job.setJobName("GenerateGraphsJob");
job.setJarByClass(GenerateGraphsJob.class);
configureJob(job);
int resultCode = job.waitForCompletion(true) ? 0 : 1;
return resultCode;
}
private void configureJob(Job job) throws IOException {
try {
Configuration config = getConf();
Path inputPath = ConfigHelper.getChartInputPath(config);
Path outputPath = ConfigHelper.getChartOutputPath(config);
job.setInputFormatClass(AvroKeyInputFormat.class);
AvroKeyInputFormat.addInputPath(job, inputPath);
AvroJob.setInputKeySchema(job, StockReport.SCHEMA$);
job.setMapperClass(StockAverageMapper.class);
job.setCombinerClass(StockAverageCombiner.class);
job.setReducerClass(StockAverageReducer.class);
FileOutputFormat.setOutputPath(job, outputPath);
} catch (IOException | ClassCastException e) {
LOG.error("An job error has occurred.", e);
}
}
映射器：
public class StockAverageMapper extends
Mapper<AvroKey<StockReport>, NullWritable, StockYearSymbolKey, StockReport> {
private static Logger LOG = LoggerFactory.getLogger(StockAverageMapper.class);
private final StockReport stockReport = new StockReport();
private final StockYearSymbolKey stockKey = new StockYearSymbolKey();
@Override
protected void map(AvroKey<StockReport> inKey, NullWritable ignore, Context context)
throws IOException, InterruptedException {
try {
StockReport inKeyDatum = inKey.datum();
for (Stock stock : inKeyDatum.getStocks()) {
updateKey(inKeyDatum, stock);
updateValue(inKeyDatum, stock);
context.write(stockKey, stockReport);
}
} catch (Exception ex) {
LOG.debug(ex.toString());
}
}
地图输出键的架构：
{
"namespace": "avro.model",
"type": "record",
"name": "StockYearSymbolKey",
"fields": [
{
"name": "year",
"type": "int"
},
{
"name": "symbol",
"type": "string"
}
]
}
堆栈跟踪：
java.lang.Exception: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
at org.apache.avro.mapreduce.AvroKeyInputFormat.createRecordReader(AvroKeyInputFormat.java:47)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:492)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:735)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
编辑：这并不重要，但我正在努力将其减少为我可以从中创建 JFreeChart 输出的数据。没有通过映射器，所以不应该相关。
问题是 org.apache.hadoop.mapreduce.TaskAttemptContext 是 class in Hadoop 1 but became an interface in Hadoop 2。
这就是依赖于 Hadoop 库的库需要为 Hadoop 1 和 Hadoop 2 分别编译 jarfile 的原因之一。根据您的堆栈跟踪，您似乎以某种方式获得了 Hadoop1 编译的 Avro jarfile , 尽管 运行 Hadoop 2.4.1.
download mirrors for Avro provide nice separate downloadables for avro-mapred-1.7.7-hadoop1.jar vs avro-mapred-1.7.7-hadoop2.jar.
问题是 Avro 1.7.7 支持 2 个版本的 Hadoop，因此依赖于两个 Hadoop 版本。默认情况下，Avro 1.7.7 jar 依赖于旧的 Hadoop 版本。
要使用 Avro 1.7.7 和 Hadoop2 构建，只需向 Maven 依赖项添加额外的 classifier 行：
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro-mapred</artifactId>
<version>1.7.7</version>
<classifier>hadoop2</classifier>
</dependency>
这将告诉 maven 搜索 avro-mapred-1.7.7-hadoop2.jar，而不是 avro-mapred-1.7.7.jar
同样适用于 Avro 1.7.4 及更高版本
到目前为止还没有找到解决我的特定问题的方法。它至少不起作用。这让我很疯狂。这个特殊的组合在 google space 中似乎没有太多。据我所知，我的错误发生在作业进入映射器时。该作业的输入是 avro 模式的输出，虽然我也尝试过未压缩，但它是用 deflate 压缩的。
阿芙罗：1.7.7 Hadoop：2.4.1
我遇到了这个错误，我不确定为什么。这是我的工作，mapper 和 reduce。映射器进入时发生错误。
示例未压缩的 Avro 输入文件（StockReport.SCHEMA 是这样定义的）
{"day": 3, "month": 2, "year": 1986, "stocks": [{"symbol": "AAME", "timestamp": 507833213000, "dividend": 10.59}]}
工作
@Override
public int run(String[] strings) throws Exception {
Job job = Job.getInstance();
job.setJobName("GenerateGraphsJob");
job.setJarByClass(GenerateGraphsJob.class);
configureJob(job);
int resultCode = job.waitForCompletion(true) ? 0 : 1;
return resultCode;
}
private void configureJob(Job job) throws IOException {
try {
Configuration config = getConf();
Path inputPath = ConfigHelper.getChartInputPath(config);
Path outputPath = ConfigHelper.getChartOutputPath(config);
job.setInputFormatClass(AvroKeyInputFormat.class);
AvroKeyInputFormat.addInputPath(job, inputPath);
AvroJob.setInputKeySchema(job, StockReport.SCHEMA$);
job.setMapperClass(StockAverageMapper.class);
job.setCombinerClass(StockAverageCombiner.class);
job.setReducerClass(StockAverageReducer.class);
FileOutputFormat.setOutputPath(job, outputPath);
} catch (IOException | ClassCastException e) {
LOG.error("An job error has occurred.", e);
}
}
映射器：
public class StockAverageMapper extends
Mapper<AvroKey<StockReport>, NullWritable, StockYearSymbolKey, StockReport> {
private static Logger LOG = LoggerFactory.getLogger(StockAverageMapper.class);
private final StockReport stockReport = new StockReport();
private final StockYearSymbolKey stockKey = new StockYearSymbolKey();
@Override
protected void map(AvroKey<StockReport> inKey, NullWritable ignore, Context context)
throws IOException, InterruptedException {
try {
StockReport inKeyDatum = inKey.datum();
for (Stock stock : inKeyDatum.getStocks()) {
updateKey(inKeyDatum, stock);
updateValue(inKeyDatum, stock);
context.write(stockKey, stockReport);
}
} catch (Exception ex) {
LOG.debug(ex.toString());
}
}
地图输出键的架构：
{
"namespace": "avro.model",
"type": "record",
"name": "StockYearSymbolKey",
"fields": [
{
"name": "year",
"type": "int"
},
{
"name": "symbol",
"type": "string"
}
]
}
堆栈跟踪：
java.lang.Exception: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
at org.apache.avro.mapreduce.AvroKeyInputFormat.createRecordReader(AvroKeyInputFormat.java:47)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:492)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:735)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
编辑：这并不重要，但我正在努力将其减少为我可以从中创建 JFreeChart 输出的数据。没有通过映射器，所以不应该相关。
问题是 org.apache.hadoop.mapreduce.TaskAttemptContext 是 class in Hadoop 1 but became an interface in Hadoop 2。
这就是依赖于 Hadoop 库的库需要为 Hadoop 1 和 Hadoop 2 分别编译 jarfile 的原因之一。根据您的堆栈跟踪，您似乎以某种方式获得了 Hadoop1 编译的 Avro jarfile , 尽管 运行 Hadoop 2.4.1.
download mirrors for Avro provide nice separate downloadables for avro-mapred-1.7.7-hadoop1.jar vs avro-mapred-1.7.7-hadoop2.jar.
问题是 Avro 1.7.7 支持 2 个版本的 Hadoop，因此依赖于两个 Hadoop 版本。默认情况下，Avro 1.7.7 jar 依赖于旧的 Hadoop 版本。
要使用 Avro 1.7.7 和 Hadoop2 构建，只需向 Maven 依赖项添加额外的 classifier 行：
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro-mapred</artifactId>
<version>1.7.7</version>
<classifier>hadoop2</classifier>
</dependency>
这将告诉 maven 搜索 avro-mapred-1.7.7-hadoop2.jar，而不是 avro-mapred-1.7.7.jar
同样适用于 Avro 1.7.4 及更高版本