在 Yarn 中找不到 spark-submit 完成的工作
Can't find spark-submit finished job in Yarn
我使用 spark-submit 提交了作业，作业正确完成但未在 Yarn 中显示。
如何在 Yarn 中直接查看进程？我已经指定了 yarn 客户端模式，但在 yarn web 控制台中我什么也看不到。
基于 ，yarn application -list -appStates FINISHED 可以胜任。同样在同一个线程中，您会发现另一个可能的原因：
It turns out that I had enabled Log aggregation in YARN but had set the yarn.nodemanager.remote-app-log-dir to a custom hdfs directory (/tmp/yarnlogs), So logs were actually getting aggregated at /tmp/yarnlogs in HDFS, but the yarn command was still searching for logs at the default location on HDFS (/tmp/logs). So changing the property to its default value fixed it for me.
希望对您有所帮助！
我使用 spark-submit 提交了作业，作业正确完成但未在 Yarn 中显示。
如何在 Yarn 中直接查看进程？我已经指定了 yarn 客户端模式，但在 yarn web 控制台中我什么也看不到。
基于 yarn application -list -appStates FINISHED 可以胜任。同样在同一个线程中，您会发现另一个可能的原因：
It turns out that I had enabled Log aggregation in YARN but had set the yarn.nodemanager.remote-app-log-dir to a custom hdfs directory (/tmp/yarnlogs), So logs were actually getting aggregated at /tmp/yarnlogs in HDFS, but the yarn command was still searching for logs at the default location on HDFS (/tmp/logs). So changing the property to its default value fixed it for me.
希望对您有所帮助！