Spark 应用程序与 Hive Metastore 同步 - "There is no primary group for UGI spark" 错误

Question

我正在运行在 Kubernetes 集群上执行一个简单的 Spark 作业，该作业使用 Hive 分类将数据写入 HDFS。无论出于何种原因，我的应用程序无法运行 Spark SQL 命令，但出现以下异常：

21/09/22 09:23:54 ERROR SplunkStreamListener: |exception=org.apache.spark.sql.AnalysisException
org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException There is no primary group for UGI spark (auth:SIMPLE));
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
    at org.apache.spark.sql.hive.HiveExternalCatalog.createDatabase(HiveExternalCatalog.scala:183)
    at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.createDatabase(ExternalCatalogWithListener.scala:47)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createDatabase(SessionCatalog.scala:211)
    at org.apache.spark.sql.execution.command.CreateDatabaseCommand.run(ddl.scala:70)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
    at org.apache.spark.sql.Dataset$$anonfun.apply(Dataset.scala:194)
    at org.apache.spark.sql.Dataset$$anonfun.apply(Dataset.scala:194)
    at org.apache.spark.sql.Dataset$$anonfun.apply(Dataset.scala:3370)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId.apply(SQLExecution.scala:80)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
    at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withAction(Dataset.scala:3369)
    at org.apache.spark.sql.Dataset.<init>(Dataset.scala:194)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:643)

我正在通过 Thrift URL 连接到 Hive Metastore。 docker 容器运行将应用程序作为非根用户。是否需要添加用户以与 Metastore 同步？

Answer 1

尝试在设置 spark 上下文之前添加这个

System.setProperty("HADOOP_USER_NAME", "root")

Spark 应用程序与 Hive Metastore 同步 - "There is no primary group for UGI spark" 错误

Spark application syncing with Hive metastore - "There is no primary group for UGI spark" error

hive

apache-spark

kubernetes

apache-spark-sql

hive-metastore