CDH 5.4.4 Oozie 无法 运行 Sqoop 操作 - ClassNotFound SqoopMain
CDH 5.4.4 Oozie unable to run Sqoop action - ClassNotFound SqoopMain
ClassNotFound SqoopMain
我正在尝试通过 Cloudera 5 上的 Oozie 运行 一个简单的 Sqoop 操作。4.x（通过他们的 QuickStart VM，我认为应该正确预配置？）
当我通过 Sqoop CLI 运行 导入命令时，一切正常。但是，当我尝试使用 Oozie 工作流（通过 Hue）运行 相同的命令时，它找不到 SqoopMain class。
错误日志
2015-07-14 14:58:02,997 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] [***0000001-150714084022371-oozie-oozi-W@sqoop-import***]Action updated in DB!
2015-07-14 14:58:12,802 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] callback for action [0000001-150714084022371-oozie-oozi-W@sqoop-import]
2015-07-14 14:58:13,058 INFO org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] action completed, external ID [job_1436888351169_0003]
2015-07-14 14:58:13,078 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exception invoking main(), java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
2015-07-14 14:58:13,085 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access0(LocalContainerLauncher.java:181)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.run(LocalContainerLauncher.java:224)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
... 13 more
工作流操作
<action name="sqoop-import">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/tmp/etl/${etlUser}/vet_product_categories"/>
</prepare>
<arg>import</arg>
<arg>--connect</arg>
<arg>jdbc:mysql://${oltpHost}/${oltpName}</arg>
<arg>--username</arg>
<arg>${oltpUser}</arg>
<arg>--password</arg>
<arg>${oltpPassword}</arg>
<arg>--table</arg>
<arg>view_et_product_categories</arg>
<arg>--target-dir</arg>
<arg>/tmp/etl/${etlUser}/vet_product_categories</arg>
<arg>--as-avrodatafile</arg>
<arg>-m</arg>
<arg>1</arg>
</sqoop>
<ok to="done"/>
<error to="fail"/>
</action>
更新 1
在 cloudera 管理器中查找 oozie.service.WorkflowAppService.system.libpath，它被设置为 /user/oozie - 它会将 share/lib 附加到您在此字段中输入的任何内容，因此完整路径是 /usr/oozie/share/lib.
HDFS 中的文件夹使用时间戳进行版本控制。 - 我不确定 oozie 如何将这些 class 添加到 class 路径，或者它是否需要额外的帮助来获取它：/user/oozie/share/lib/lib_20150609033900
事实证明，您需要提供 job.properties 和 oozie.use.system.libpath=true
看来把这个放在workflow.xml里是不行的。
基本上，我现在拥有的是一个没有配置的裸 sqoop 操作和一个具有我需要的所有属性的 job.properties。
ClassNotFound SqoopMain
我正在尝试通过 Cloudera 5 上的 Oozie 运行 一个简单的 Sqoop 操作。4.x（通过他们的 QuickStart VM，我认为应该正确预配置？）
当我通过 Sqoop CLI 运行 导入命令时，一切正常。但是，当我尝试使用 Oozie 工作流（通过 Hue）运行 相同的命令时，它找不到 SqoopMain class。
错误日志
2015-07-14 14:58:02,997 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] [***0000001-150714084022371-oozie-oozi-W@sqoop-import***]Action updated in DB!
2015-07-14 14:58:12,802 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] callback for action [0000001-150714084022371-oozie-oozi-W@sqoop-import]
2015-07-14 14:58:13,058 INFO org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] action completed, external ID [job_1436888351169_0003]
2015-07-14 14:58:13,078 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exception invoking main(), java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
2015-07-14 14:58:13,085 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access0(LocalContainerLauncher.java:181)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.run(LocalContainerLauncher.java:224)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
... 13 more
工作流操作
<action name="sqoop-import">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/tmp/etl/${etlUser}/vet_product_categories"/>
</prepare>
<arg>import</arg>
<arg>--connect</arg>
<arg>jdbc:mysql://${oltpHost}/${oltpName}</arg>
<arg>--username</arg>
<arg>${oltpUser}</arg>
<arg>--password</arg>
<arg>${oltpPassword}</arg>
<arg>--table</arg>
<arg>view_et_product_categories</arg>
<arg>--target-dir</arg>
<arg>/tmp/etl/${etlUser}/vet_product_categories</arg>
<arg>--as-avrodatafile</arg>
<arg>-m</arg>
<arg>1</arg>
</sqoop>
<ok to="done"/>
<error to="fail"/>
</action>
更新 1
在 cloudera 管理器中查找 oozie.service.WorkflowAppService.system.libpath，它被设置为 /user/oozie - 它会将 share/lib 附加到您在此字段中输入的任何内容，因此完整路径是 /usr/oozie/share/lib.
HDFS 中的文件夹使用时间戳进行版本控制。 - 我不确定 oozie 如何将这些 class 添加到 class 路径，或者它是否需要额外的帮助来获取它：/user/oozie/share/lib/lib_20150609033900
事实证明，您需要提供 job.properties 和 oozie.use.system.libpath=true
看来把这个放在workflow.xml里是不行的。
基本上，我现在拥有的是一个没有配置的裸 sqoop 操作和一个具有我需要的所有属性的 job.properties。