Hadoop:Cascading 流异常

Question

我安装了 hadoop 1.0.4 和 hive 0.12.When 我运行级联模式在此它给出级联流异常。当我运行使用以下 hadoop 命令时

hadoop jar bulid/libs/pattern-example*.jar

我遇到了上述异常，为了参考，我包含了级联代码。

Tap inputTap = new Hfs(new TextDelimited(true, "\t"),
                    "hdfs://hdmaster:54310/user/hive/warehouse/temp/Dataformated/finalformated");
String classifyPath=Output Path;
hdfsPath = classifyPath/pmml File Name;
            Tap classifyTap = new Hfs(new TextDelimited(true, "\t"),
                    classifyPath/pmml File Name));
            String formatLocalHdfsData = classifyPath/PMML FILE NAME);
            FlowDef flowDef = FlowDef.flowDef().setName("classify")
                    .addSource("input", inputTap)// input is LFs or HFS
                    .addSink("classify", classifyTap);
            flowDef.addAssemblyPlanner(pmmlPlanner);
            Flow classifyFlow = flowConnector.connect(flowDef);
            classifyFlow.writeDOT("dot/classify.dot");
            classifyFlow.complete();

级联流异常

Exception in thread "main" cascading.flow.FlowException: step failed: (1/1) ...eg_Nocoerce20150513093050, with job id: job_201505130921_0003, please see cluster logs for failure messages
at cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:221)
at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:149)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:124)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:43)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)

日志文件异常

java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: cascading.flow.FlowException: internal error during mapper configuration
at cascading.flow.hadoop.FlowMapper.configure(FlowMapper.java:99)
... 14 more
Caused by: java.io.InvalidClassException: cascading.tap.hadoop.Hfs; local class incompatible: stream classdesc serialVersionUID = -2723557385578774808, local class serialVersionUID = -4246440312226820384
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:560)

请帮我解决这个问题。

Answer 1

我解决了这个问题。在日志文件中，我遇到了 serialVersionID 兼容性问题。生成新的 SerialVersionID 并且它起作用了。

Hadoop:Cascading 流异常

Hadoop:Cascading FlowException

hadoop

hive

cascading