PigUnit 与 Hadoop 2.x

Question

我正在尝试将 PigUnit 与 Hadoop 2.4.1 结合使用。我有一个非常简单的 Pig 脚本，我想在 PigTest 中执行它，但是当我运行这个测试时，我得到以下异常：

java.lang.InstantiationError: org.apache.hadoop.mapreduce.JobContext
    at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.createJobContext(HadoopShims.java:66)
    at org.apache.pig.backend.hadoop.executionengine.fetch.FetchPOStoreImpl.createStoreFunc(FetchPOStoreImpl.java:58)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:103)
    at org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.init(FetchLauncher.java:121)
    at org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.launchPig(FetchLauncher.java:78)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:275)
    at org.apache.pig.PigServer.launchPlan(PigServer.java:1367)
    at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1352)
    at org.apache.pig.PigServer.storeEx(PigServer.java:1011)
    at org.apache.pig.PigServer.store(PigServer.java:974)
    at org.apache.pig.PigServer.openIterator(PigServer.java:887)
    at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:752)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:228)
    at org.apache.pig.pigunit.pig.PigServer.registerScript(PigServer.java:55)
    at org.apache.pig.pigunit.PigTest.registerScript(PigTest.java:170)
    at org.apache.pig.pigunit.PigTest.assertOutput(PigTest.java:242)
    at com.mongodb.hadoop.pig.PigTest.testSimplePigScript(PigTest.java:57)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at org.junit.runners.model.FrameworkMethod.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access[=11=]0(ParentRunner.java:53)
    at org.junit.runners.ParentRunner.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)

在 Hadoop 2.4.1 中，org.apache.hadoop.mapreduce.JobContext 是一个接口，而不是 class，所以在我看来 PigUnit 似乎在某种程度上期待旧版本的 Hadoop？如果确实如此，我如何将 PigUnit 指向更新的 Hadoop 罐子？

Answer 1

原来是我用错了Pig版本。 Pig 在 Maven 上有多个工件，我需要在我的 build.gradle 中提供一个分类器，以便指定适用于 Hadoop 的 Pig jar 2.x:

compile "org.apache.pig:pig:${pigVersion}:h2"

帮助我解决了 Gradle/dependency 问题。

PigUnit 与 Hadoop 2.x

PigUnit with Hadoop 2.x

java

hadoop

classpath

intellij-idea

apache-pig