EMR 步骤中的 Spark Application Level 日志

Question

我在 EMR 步骤中运行 spark 应用程序，但作业由于某些错误而失败，我想查看该错误。我已经检查了 stderr 但它没有提供任何关于错误的详细信息。据说

Exception in thread "main" org.apache.spark.SparkException: Application application_1593934145491_0002 finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1149)
    at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1526)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:853)
    at org.apache.spark.deploy.SparkSubmit.doRunMain(SparkSubmit.scala:161)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon.doSubmit(SparkSubmit.scala:928)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:937)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/07/05 07:50:37 INFO ShutdownHookManager: Shutdown hook called

有人可以帮我吗？我想查看应用程序级别的日志。

Answer 1

它应该总是在 /container 下，但如果找不到它，请尝试 ssh 主节点和运行 spark-submit

Answer 2

在客户端启用调试模式和运行脚本后，我能够在 Steps/Step_ID/stdout.gz

中看到 Spark 应用程序级别的日志

EMR 步骤中的 Spark Application Level 日志

Spark Application Level logs in EMR step

apache-spark

pyspark

amazon-emr