AWS Glue：命令失败，错误代码为 1

Question

几周来我一直在与这个错误作斗争。我尝试了许多不同的应用程序，但找不到与此错误的任何一致性。有时，如果我将作业书签设置更改为暂停、启用或恢复为禁用，它就会开始工作。我有一个 java jar，我正在引用胶水作业，我正在从中调用一些方法。有时，如果我重建工件，作业就会开始工作并且不再抛出此错误。我有另一份工作使用完全相同的 jar 并且永远不会抛出错误。我曾尝试创建一份新工作以重新开始，但我遇到了同样的问题。这是它在日志中给出的错误堆栈。应用程序的代码是一个简单的数据帧，从 S3 读取并写入 s3 中的另一个位置。

val df = spark.read.parquet(source)

df.write.mode("overwrite").format("parquet").save(destination)

错误堆栈：

18/04/30 14:40:35 ERROR ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: com.amazonaws.services.glue.AWSGlue.getJobBookmark(Lcom/amazonaws/services/glue/model/GetJobBookmarkRequest;)Lcom/amazonaws/services/glue/model/GetJobBookmarkResult;
java.lang.NoSuchMethodError: com.amazonaws.services.glue.AWSGlue.getJobBookmark(Lcom/amazonaws/services/glue/model/GetJobBookmarkRequest;)Lcom/amazonaws/services/glue/model/GetJobBookmarkResult;
at com.amazonaws.services.glue.util.JobBookmarkServiceShim$$anonfun.apply(JobBookmarkUtils.scala:54)
at com.amazonaws.services.glue.util.JobBookmarkServiceShim$$anonfun.apply(JobBookmarkUtils.scala:54)
at scala.util.Try$.apply(Try.scala:192)
at com.amazonaws.services.glue.util.JobBookmarkServiceShim.<init>(JobBookmarkUtils.scala:54)
at com.amazonaws.services.glue.util.JobBookmark$.configure(JobBookmarkUtils.scala:178)
at com.amazonaws.services.glue.util.Job$.init(Job.scala:68)
at com.amazonaws.services.glue.util.Job$.init(Job.scala:32)
at NetezzaRawToRefined$.main(script_2018-04-30-14-39-54.scala:16)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.amazonaws.services.glue.util.GlueExceptionWrapper$$anonfun.apply$mcV$sp(GlueExceptionWrapper.scala:29)
at com.amazonaws.services.glue.util.GlueExceptionWrapper$$anonfun.apply(GlueExceptionWrapper.scala:29)
at com.amazonaws.services.glue.util.GlueExceptionWrapper$$anonfun.apply(GlueExceptionWrapper.scala:29)
at scala.util.Try$.apply(Try.scala:192)
at com.amazonaws.services.glue.util.GlueExceptionWrapper$.delayedEndpoint$com$amazonaws$services$glue$util$GlueExceptionWrapper(GlueExceptionWrapper.scala:28)
at com.amazonaws.services.glue.util.GlueExceptionWrapper$delayedInit$body.apply(GlueExceptionWrapper.scala:11)
at scala.Function0$
class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main.apply(App.scala:76)
at scala.App$$anonfun$main.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at com.amazonaws.services.glue.util.GlueExceptionWrapper$.main(GlueExceptionWrapper.scala:11)
at com.amazonaws.services.glue.util.GlueExceptionWrapper.main(GlueExceptionWrapper.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon.run(ApplicationMaster.scala:635)
18/04/30 14:40:35 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: com.amazonaws.services.glue.AWSGlue.getJobBookmark(Lcom/amazonaws/services/glue/model/GetJobBookmarkRequest;)Lcom/amazonaws/services/glue/model/GetJobBookmarkResult;)

Answer 1

java.lang.NoSuchMethodError: com.amazonaws.services.glue.AWSGlue.getJobBookmark(Lcom/amazonaws/services/glue/model/GetJobBookmarkRequest;)Lcom/amazonaws/services/glue/model/GetJobBookmarkResult;

如果您的运行时间的 jar 版本与打包版本不同，通常会发生此错误。我建议确保您打包的是相同版本的 jar。

AWS Glue：命令失败，错误代码为 1

AWS Glue: Command failed with error code 1

amazon-web-services

aws-glue