Spark Thrift 服务器 - 无法启动（Hive 错误）

Question

我正在尝试使用 Hive 0.14（最新的 HortonWorks 发行版）在 HDP 2.2.0 上启动 Spark Thrift 服务器 (Spark 1.2)，但不幸的是，我遇到了这个对我来说非常神秘的错误：

Exception in thread "main" java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "5s"
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
        at org.apache.spark.sql.hive.HiveContext$$anonfun.apply(HiveContext.scala:235)
        at org.apache.spark.sql.hive.HiveContext$$anonfun.apply(HiveContext.scala:231)
        at scala.Option.orElse(Option.scala:257)
        at org.apache.spark.sql.hive.HiveContext.x$lzycompute(HiveContext.scala:231)
        at org.apache.spark.sql.hive.HiveContext.x(HiveContext.scala:229)
        at org.apache.spark.sql.hive.HiveContext.hiveconf$lzycompute(HiveContext.scala:229)
        at org.apache.spark.sql.hive.HiveContext.hiveconf(HiveContext.scala:229)
        at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:292)
        at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:276)
        at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:248)
        at org.apache.spark.sql.SQLContext$$anonfun.apply(SQLContext.scala:91)
        at org.apache.spark.sql.SQLContext$$anonfun.apply(SQLContext.scala:90)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
        at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:90)
        at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:72)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:51)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:56)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NumberFormatException: For input string: "5s"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:492)
        at java.lang.Integer.parseInt(Integer.java:527)
        at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1094)
        at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1211)
        at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1220)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:58)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
        ... 26 more

有谁能帮我解决这个问题吗？当我尝试从与 Hive 相关的 Spark shell 执行任何命令时，也会发生同样的情况（比如创建 table、将数据插入现有命令等 ..）

Answer 1

您应该检查您的配置。

检查您是否可以从 Hive 发出命令。
如果没有，在配置单元中查找字符串“5s”-site.xml

Answer 2

我遇到过同样的BUG。 HDP2.2.4 有一个错误的 hive 配置文件

<property>
  <name>hive.metastore.client.connect.retry.delay</name>
  <value>5s</value>
</property>

应该是

<value>5<values>

Hontonworks 论坛对于来自中国的用户来说非常慢。所以我不能 post 这个 bug 给他们 =_="

Spark Thrift 服务器 - 无法启动（Hive 错误）

Spark Thrift Server - Unable to start (Hive error)

apache-spark

hive

hortonworks-data-platform

spark-thriftserver