如何在 MRUnit 中使用 avro 参数测试减速器？

Question

我正在尝试测试下面的 Reducer 波纹管

Reducer<CompositeKeyWritable, AvroValue<Class1>, AvroKey<Class2>, NullWritable>

出于这些目的，我在测试中使用了下一个代码：

CustomReducer reducer = new CustomReducer();
    reduceDriver = new ReduceDriver<CompositeKeyWritable, AvroValue<Class1>, AvroKey<Class2>, NullWritable>();

    Job job = new Job();

    reduceDriver = ReduceDriver.newReduceDriver(reducer).withConfiguration(job.getConfiguration());
    reduceDriver.setReducer(reducer);

    job.setInputFormatClass(AvroKeyInputFormat.class);

    job.setMapOutputKeyClass(CompositeKeyWritable.class);
    AvroJob.setMapOutputValueSchema(job, union);

    AvroJob.setOutputKeySchema(job, Class2.SCHEMA$);
    job.setOutputFormatClass(AvroKeyOutputFormat.class);
    job.setOutputValueClass(NullWritable.class);

结果我得到了下一个错误：

java.lang.RuntimeException: java.io.EOFException
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:86)
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:110)
at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:675)
at org.apache.hadoop.mrunit.ReduceDriverBase.addInput(ReduceDriverBase.java:167)
at org.apache.hadoop.mrunit.ReduceDriverBase.addInput(ReduceDriverBase.java:181)
at com.wellcare.caregap.hadoop.reducers.CareGapReducerTest.testReducer(CareGapReducerTest.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access[=12=]0(ParentRunner.java:58)
at org.junit.runners.ParentRunner.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

作业配置有什么问题？如果有人帮助我解决这些异常，我将非常感激。

Answer 1

调试您的方法 readFields 并写入 Class1。当您使用 reduceDriver.withInput(yourkey, yourvalue);

添加输入时，它似乎在到达其中之一的 DataInput 或 DataOutput 流的末尾后尝试读取或写入字段

如何在 MRUnit 中使用 avro 参数测试减速器？

How to test reducer with avro params in MRUnit?

hadoop

mapreduce

bigdata

avro

mrunit