Sqoop Export Oozie 工作流因找不到文件而失败，在运行来自控制台时有效

Question

我有一个有 6 个节点的 hadoop 集群。我正在通过 Sqoop 从 MSSQL 中提取数据并返回到 MSSQL 中。 Sqoop 导入命令运行良好，我可以从控制台（在其中一个 hadoop 节点上）运行执行 sqoop 导出命令。这是我运行的 shell 脚本：

SQLHOST=sqlservermaster.local
SQLDBNAME=db1
HIVEDBNAME=db1
BATCHID=
USERNAME="sqlusername"
PASSWORD="password"


sqoop export --connect 'jdbc:sqlserver://'$SQLHOST';username='$USERNAME';password='$PASSWORD';database='$SQLDBNAME'' --table ExportFromHive --columns col1,col2,col3 --export-dir /apps/hive/warehouse/$HIVEDBNAME.db/hivetablename

当我从 oozie 工作流中运行此命令并传递相同的参数时，我收到错误（当从 yarn scheduler 屏幕挖掘实际作业运行日志时）：

**2015-10-01 20:55:31,084 WARN [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job init failed
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://hadoopnode1:8020/user/root/.staging/job_1443713197941_0134/job.splitmetainfo
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1568)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1432)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1390)
    at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
    at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
    at org.apache.hadoop.yarn.state.StateMachineFactory.access0(StateMachineFactory.java:46)
    at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1312)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1080)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.run(MRAppMaster.java:1519)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoopnode1:8020/user/root/.staging/job_1443713197941_0134/job.splitmetainfo
    at org.apache.hadoop.hdfs.DistributedFileSystem.doCall(DistributedFileSystem.java:1309)
    at org.apache.hadoop.hdfs.DistributedFileSystem.doCall(DistributedFileSystem.java:1301)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
    at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1563)
    ... 17 more**

有没有人见过这个问题并且能够解决它？它只发生在 oozie 工作流程中。有类似的主题，但似乎没有人解决了这个特定问题。

谢谢！

Answer 1

我能够通过在 job.properties 文件上为用户 yarn 设置 oozie 工作流的 user.name 属性来解决这个问题。

user.name=yarn

我认为问题在于它没有在 /user/root 下创建暂存文件的权限。一旦我将运行用户修改为 yarn，暂存文件就在 /user/yarn 下创建，它确实具有适当的权限。

Sqoop Export Oozie 工作流因找不到文件而失败，在运行来自控制台时有效

Sqoop Export Oozie Workflow Fails with File Not Found, Works when ran from the console

hadoop

hive

sqoop

hue

oozie

Sqoop Export Oozie 工作流因找不到文件而失败，在 运行 来自控制台时有效

Sqoop Export Oozie Workflow Fails with File Not Found, Works when ran from the console

hadoop

hive

sqoop

hue

oozie

Sqoop Export Oozie 工作流因找不到文件而失败，在运行来自控制台时有效