java.lang.ClassCastException: org.apache.hadoop.io.serializer.WritableSerialization 无法转换为 org.apache.hadoop.io.serializer.Serialization

Question

我正在使用 MRUnit 测试 MultipleOutputs。测试用例失败并显示以下消息。

java.lang.ClassCastException: org.apache.hadoop.io.serializer.WritableSerialization cannot be cast to org.apache.hadoop.io.serializer.Serialization

我根据我看到的示例使用以下注释。

@RunWith(PowerMockRunner.class)
@PrepareForTest({MultipleOutputs.class, LoadMapper.class, LoadReducer.class})

作为参考，我添加了用于测试的代码。

@RunWith(PowerMockRunner.class)
@PrepareForTest({MultipleOutputs.class, LoadMapper.class, LoadReducer.class})
@PowerMockIgnore({
    "org.apache.log4j.*",
    "javax.xml.*",
    "org.xml.sax.*", 
    "org.apache.xerces.*",
    "org.w3c.dom.*"
    })
public class LoadTest extends LoadReducer {

    MapDriver<LongWritable, Text, NullWritable, Text> mapDriver;
    Configuration conf = new Configuration();

    @SuppressWarnings("deprecation")
    @Before
    public void setUp() {
        conf.set(LoadConstants.START_TAG_KEY, LoadConstants.START_TAG_KEY);
        conf.set(LoadConstants.END_TAG_KEY, LoadConstants.END_TAG_KEY);
        LoadMapper loadMapper = new LoadMapper();
        mapDriver = MapDriver.newMapDriver(loadMapper);
        mapDriver.setConfiguration(conf);
    }

    @Test
    public void testFailure() throws IOException {
        String inputRecord = "";
        String errorMsg = "";

        mapDriver.withInput(new LongWritable(0), new Text(inputRecord))
            .withMultiOutput(LoadConstants.BAD, NullWritable.get(), new Text(inputRecord + errorMsg))
            .runTest();

    }
}

错误的堆栈跟踪如下：

java.lang.LinkageError: loader constraint violation: when resolving method "org.apache.hadoop.io.serializer.SerializationFactory.<init>(Lorg/apache/hadoop/conf/Configuration;)V" the class loader (instance of org/powermock/core/classloader/MockClassLoader) of the current class, org/apache/hadoop/mrunit/internal/io/Serialization, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class, org/apache/hadoop/io/serializer/SerializationFactory, have different Class objects for the type org/apache/hadoop/conf/Configuration used in the signature
    at org.apache.hadoop.mrunit.internal.io.Serialization.<init>(Serialization.java:39)
    at org.apache.hadoop.mrunit.TestDriver.getSerialization(TestDriver.java:530)
    at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:675)
    at org.apache.hadoop.mrunit.TestDriver.copyPair(TestDriver.java:679)
    at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:120)
    at org.apache.hadoop.mrunit.MapDriverBase.withInput(MapDriverBase.java:210)
    at x.x.x.LoadTest.testFailure(LoadTest.java:20)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:66)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:310)
    at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:86)
    at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:94)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.executeTest(PowerMockJUnit44RunnerDelegateImpl.java:294)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTestInSuper(PowerMockJUnit47RunnerDelegateImpl.java:127)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:82)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:282)
    at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:84)
    at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:49)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:207)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:146)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.run(PowerMockJUnit44RunnerDelegateImpl.java:120)
    at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
    at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.run(PowerMockJUnit44RunnerDelegateImpl.java:118)
    at org.powermock.modules.junit4.common.internal.impl.JUnit4TestSuiteChunkerImpl.run(JUnit4TestSuiteChunkerImpl.java:101)
    at org.powermock.modules.junit4.common.internal.impl.AbstractCommonPowerMockRunner.run(AbstractCommonPowerMockRunner.java:53)
    at org.powermock.modules.junit4.PowerMockRunner.run(PowerMockRunner.java:53)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

为 MRUnit 添加了 POM 依赖项：

<dependency>
    <groupId>org.apache.mrunit</groupId>
    <artifactId>mrunit</artifactId>
    <version>1.1.0</version>
    <classifier>hadoop2</classifier>
</dependency>

Hadoop version: 2.4.1 Environment : Windows 7 - 64 bit

如果有人遇到过这个问题，请帮忙。

Answer 1

从测试 class 中删除 mapDriver.setConfiguration 后，问题得到解决。

 @SuppressWarnings("deprecation")
    @Before
    public void setUp() {
        conf.set(LoadConstants.START_TAG_KEY, LoadConstants.START_TAG_KEY);
        conf.set(LoadConstants.END_TAG_KEY, LoadConstants.END_TAG_KEY);
        LoadMapper loadMapper = new LoadMapper();
        mapDriver = MapDriver.newMapDriver(loadMapper);
        mapDriver.setConfiguration(conf); // Removed this line
    }

我不清楚原因。

java.lang.ClassCastException: org.apache.hadoop.io.serializer.WritableSerialization 无法转换为 org.apache.hadoop.io.serializer.Serialization

java.lang.ClassCastException: org.apache.hadoop.io.serializer.WritableSerialization cannot be cast to org.apache.hadoop.io.serializer.Serialization

hadoop

mrunit

multipleoutputs