如何检查纱线日志应用程序 ID

Question

我正在尝试运行将 bash 脚本运行 spark-submit 和运行 pyspark 脚本，但没有成功。我想使用“yarn logs -applicationId”检查纱线日志。我的问题是如何找到合适的应用程序 ID？

以下是我得到的部分错误

Answer 1

1. Using Yarn Logs:

在日志中您可以看到 tracking URL: http://<nn>:8088/proxy/application_*****/

如果您复制并打开 link 您可以在 Resourcemanager 中看到该应用程序的所有日志。

2.Using Spark application:

从sparkContext我们可以得到applicationID。

print(spark.sparkContext.aplicationId)

3. Using yarn application command:

使用yarn application --list命令获取集群上的所有运行 yarn应用然后使用

yarn application --help
-appStates <States>             Works with -list to filter applications
                                 based on input comma-separated list of
                                 application states. The valid application
                                 state can be one of the following:
                                 ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
                                 NING,FINISHED,FAILED,KILLED
 -appTypes <Types>               Works with -list to filter applications
                                 based on input comma-separated list of
                                 application types.
 -help                           Displays help for all commands.
 -kill <Application ID>          Kills the application.
 -list                           List applications. Supports optional use
                                 of -appTypes to filter applications based
                                 on application type, and -appStates to
                                 filter applications based on application
                                 state.
 -movetoqueue <Application ID>   Moves the application to a different
                                 queue.
 -queue <Queue Name>             Works with the movetoqueue command to
                                 specify which queue to move an
                                 application to.
 -status <Application ID>        Prints the status of the application.

List all the finished applications:

yarn application -appStates FINISHED -list

Answer 2

您还可以使用 curl 通过 YARN Rest API 获取您的应用程序所需的详细信息。

state="RUNNING" // RUNNING, FAILED, COMPLETED.

user="" // userid from where you started job.

applicationTypes="spark" // Type of application

applicationName="<application_name>" // Your application name

url="http://<host_name>:8088/ws/v1/cluster/apps?state=${state}&user=${user}&applicationTypes=${applicationTypes}" // Build Rest API

applicationId=$(curl "${url}" | python -m json.tool | jq -r '.apps."app" | .[] | select(.name | contains('\"${applicationName}\"')) | .id')

输出

> echo $applicationId
application_1593019621736_42096

如何检查纱线日志应用程序 ID

How to check yarn logs application id

apache-spark

hadoop

pyspark

hadoop-yarn