由于 ZooKeeperPersistenceEngine 中的异常导致 Spark 提交失败

Question

由于 ZooKeeperPersistenceEngine 异常导致 Spark 提交失败。在我的本地环境中测试了 jar 部署，但是当我尝试将其推送到集群时它失败了。

星火：2.4.0
动物园管理员：3.4.14
斯卡拉：2.11

./spark-submit --class com.test.Main --deploy-mode cluster --master spark://10.1x.1xx.xx:27077 /tmp/test_jars/stp.jar

19/09/10 18:52:16 INtestClientCnxn: Session establishment complete on server 10.1x.1xx.xx:2181, sessionid = 0x10020111fa00001, negotiated timeout = 40000
19/09/10 18:52:16 INFO ConnectionStateManager: State change: CONNECTED
19/09/10 18:52:18 INFO Master: Registering worker 10.1x.1xx.xx:28088 with 1 cores, 4.0 GB RAM
19/09/10 18:52:25 INFO Master: Registering worker 10.1x.1xx.xx:28088 with 1 cores, 4.0 GB RAM
19/09/10 18:52:32 INFO Master: Registering worker 10.1x.1xx.xx:28088 with 1 cores, 4.0 GB RAM
19/09/10 18:52:39 INFO Master: Registering worker 10.1x.1xx.xx:28088 with 1 cores, 4.0 GB RAM
19/09/10 18:52:42 INFO Master: Registering worker 10.1x.1xx.xx:28088 with 1 cores, 4.0 GB RAM
19/09/10 18:52:46 INFO Master: Registering worker 10.1x.1xx.xx:28088 with 1 cores, 4.0 GB RAM
19/09/10 18:52:49 INFO ZooKeeperLeaderElectionAgent: We have gained leadership
19/09/10 18:52:49 WARN ZooKeeperPersistenceEngine: Exception while reading persisted file, deleting
java.io.InvalidClassException: org.apache.spark.deploy.master.WorkerInfo; local class incompatible: stream classdesc serialVersionUID = 7036178881785247710, local class serialVersionUID = -427782398096880289
        at java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
        at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1903)
        at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1772)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:108)
        at org.apache.spark.deploy.master.ZooKeeperPersistenceEngine.org$apache$spark$deploy$master$ZooKeeperPersistenceEngine$$deserializeFromFile(ZooKeeperPersistenceEngine.scala:71)
        at org.apache.spark.deploy.master.ZooKeeperPersistenceEngine$$anonfun$read.apply(ZooKeeperPersistenceEngine.scala:54)
        at org.apache.spark.deploy.master.ZooKeeperPersistenceEngine$$anonfun$read.apply(ZooKeeperPersistenceEngine.scala:54)
        at scala.collection.TraversableLike$$anonfun$flatMap.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap.apply(TraversableLike.scala:241)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at org.apache.spark.deploy.master.ZooKeeperPersistenceEngine.read(ZooKeeperPersistenceEngine.scala:54)
        at org.apache.spark.deploy.master.PersistenceEngine$$anonfun$readPersistedData.apply(PersistenceEngine.scala:87)
        at org.apache.spark.deploy.master.PersistenceEngine$$anonfun$readPersistedData.apply(PersistenceEngine.scala:86)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
        at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:320)
        at org.apache.spark.deploy.master.PersistenceEngine.readPersistedData(PersistenceEngine.scala:86)
        at org.apache.spark.deploy.master.Master$$anonfun$receive.applyOrElse(Master.scala:221)
        at org.apache.spark.rpc.netty.Inbox$$anonfun$process.apply$mcV$sp(Inbox.scala:117)
        at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
        at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:221)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:835)
19/09/10 18:52:49 WARN ZooKeeperPersistenceEngine: Exception while reading persisted file, deleting

Answer 1

清除了 zookeeper 创建的临时文件夹，问题得到解决。由于 RPC 环境设置

，工作节点也出现连接错误

由于 ZooKeeperPersistenceEngine 中的异常导致 Spark 提交失败

Spark submit failed due to exception in ZooKeeperPersistenceEngine

apache-spark

apache-zookeeper

spark-structured-streaming