通过 Oozie HDP 2.1 的作业未创建 job.splitmetainfo

Question

当尝试执行一个 sqoop 作业时，我的 Hadoop 程序作为 jar 文件传递到 -jarFiles 参数中，执行失败并出现以下错误。似乎没有任何解决方案。具有相同 Hadoop 用户的其他作业正在成功执行。

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://sandbox.hortonworks.com:8020/user/root/.staging/job_1423050964699_0003/job.splitmetainfo
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1541)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1396)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1363)
    at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
    at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
    at org.apache.hadoop.yarn.state.StateMachineFactory.access0(StateMachineFactory.java:46)
    at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:976)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:135)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1241)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1041)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.run(MRAppMaster.java:1452)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1448)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1381)

Answer 1

所以这就是我解决它的方法。我们正在使用 CDH5 到运行 Camus 从 kafka 中提取数据。我们运行负责使用命令行从 kafka 获取数据的 CamusJob:

hadoop jar...

问题是新主机没有得到所谓的"yarn-gateway"。 Cloudera 命名与服务相关的配置包并复制到 /etc/hadoop/conf 作为 "gateway"。所以我只是在 CM UI 中单击了“deploy client configuration”。 YARN 客户端 conf 已复制到每个 YARN NodeManager 节点并解决了问题。

通过 Oozie HDP 2.1 的作业未创建 job.splitmetainfo

Job via Oozie HDP 2.1 not creating job.splitmetainfo

hadoop

mapreduce

sqoop

oozie

hortonworks-data-platform