无法在 oozie 中运行 sqoop 作业（无枚举常量 com.cloudera.sqoop.SqoopOptions.FileLayout.ParquetFile）

Question

我成功创建并执行了 sqoop 导入作业，但无法在 oozie 工作流中运行它。 Sqoop 将数据从 RDBMS 导入到 HDFS 中的 parquet 文件。看来问题与镶木地板格式有关。如果我使用 --as-textfile，工作流程运行没有任何问题。

此外，我已将所有 parquet-*.jar 从 SQOOP_HOME/lib 复制到 oozie 共享库。

Sqoop 1.4.7 和 Oozie 4.3.1。

Sqoop 作业定义

$ sqoop job --create ingest_amsp_custmaster -- import --connect "jdbc:oracle:thin:@<IP>:<PORT>/<SID>" \
--username <USER> -P \
--table CUSTMASTER \
--as-parquetfile \
--target-dir /warehouse/raw/amsp/custmaster \
--delete-target-dir \
-m 1

这是我从错误日志中得到的：

java.lang.IllegalArgumentException: No enum constant com.cloudera.sqoop.SqoopOptions.FileLayout.ParquetFile
        at java.lang.Enum.valueOf(Enum.java:238)
        at org.apache.sqoop.SqoopOptions.loadProperties(SqoopOptions.java:522)
        at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:299)
        at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:198)
        at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
        at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:200)
        at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:183)
        at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:64)
        at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:51)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:436)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:350)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access0(LocalContainerLauncher.java:211)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.run(LocalContainerLauncher.java:254)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Intercepting System.exit(1)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

如有任何帮助，我们将不胜感激。

谢谢。汤田.

Answer 1

检查 Oozie 工作流中 Sqoop 的版本。好像不是 1.4.7: stack trace line

org.apache.sqoop.SqoopOptions.loadProperties(SqoopOptions.java:522)

是对应于https://github.com/apache/sqoop/blob/20af67ef60096b17e1d9585670e5ec787eb760e2/src/java/org/apache/sqoop/SqoopOptions.java#L522

无法在 oozie 中 运行 sqoop 作业（无枚举常量 com.cloudera.sqoop.SqoopOptions.FileLayout.ParquetFile）

Unable run sqoop job in oozie (No enum constant com.cloudera.sqoop.SqoopOptions.FileLayout.ParquetFile)

hadoop

sqoop

oozie

parquet

无法在 oozie 中运行 sqoop 作业（无枚举常量 com.cloudera.sqoop.SqoopOptions.FileLayout.ParquetFile）