Spark 永远不会停止处理第一批
Spark never stops processing first batch
我正在尝试 运行 我在 github 上找到的一个应用程序，这个：https://github.com/CSIRT-MU/AIDA-Framework
我 运行 在 Ubuntu 18.04.1 虚拟机中安装它。在其数据处理管道中的某个时刻，它使用了 spark，并且似乎卡在了这一点上。我可以从网络上看到 UI 我发送到那里的一些数据被批量接收。但是，它似乎永远不会完成第一批处理（即使其中有 0 条记录）。不幸的是，我没有使用 spark 的经验，也不知道到底是什么失败了。在寻找修复程序时，我遇到了可能没有足够的核心供所有执行者使用的建议。我试图将内核增加到 3 个，但这没有帮助。
我已经提供了网络上的所有屏幕UI，我希望他们能足够清楚地说明问题。有谁知道我在这里做错了什么？
截图：Spark 1 Spark 2 Spark 3 Spark 4 Spark 5 Spark 6
排队和未完成的批处理作业的输出是
callForeachRDD at NativeMethodAccessorImpl.java:0
org.apache.spark.streaming.api.python.PythonDStream.callForeachRDD(PythonDStream.scala)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
py4j.Gateway.invoke(Gateway.java:282)
py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
py4j.commands.CallCommand.execute(CallCommand.java:79)
py4j.GatewayConnection.run(GatewayConnection.java:238)
java.lang.Thread.run(Thread.java:748)
编辑：
我注意到在进程启动时会记录错误。我现在才意识到这一点，因为这个过程不会停止。错误是：
May 11 18:13:04 aida bash[5323]: 20/05/11 18:13:04 ERROR Utils: Uncaught exception in thread stdout writer for python3
May 11 18:13:04 aida bash[5323]: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]: at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]: at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRunner$$anon.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: Exception in thread "stdout writer for python3" java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]: at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]: at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRunner$$anon.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: 20/05/11 18:13:04 ERROR Utils: Uncaught exception in thread stdout writer for python3
May 11 18:13:04 aida bash[5323]: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]: at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]: at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRunner$$anon.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: Exception in thread "stdout writer for python3" java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]: at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]: at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRunner$$anon.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: 20/05/11 18:13:04 ERROR Utils: Uncaught exception in thread stdout writer for python3
May 11 18:13:04 aida bash[5323]: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]: at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]: at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRunner$$anon.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: Exception in thread "stdout writer for python3" java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]: at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]: at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRunner$$anon.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
谁能帮我解决这些错误？
Kafka 与 spark jars 的依赖关系存在冲突...
所以要么不使用 lz 压缩，要么使用 snappy 压缩，它会起作用
或按照答案 解决冲突的 jar。
我正在尝试 运行 我在 github 上找到的一个应用程序，这个：https://github.com/CSIRT-MU/AIDA-Framework
我 运行 在 Ubuntu 18.04.1 虚拟机中安装它。在其数据处理管道中的某个时刻，它使用了 spark，并且似乎卡在了这一点上。我可以从网络上看到 UI 我发送到那里的一些数据被批量接收。但是，它似乎永远不会完成第一批处理（即使其中有 0 条记录）。不幸的是，我没有使用 spark 的经验，也不知道到底是什么失败了。在寻找修复程序时，我遇到了可能没有足够的核心供所有执行者使用的建议。我试图将内核增加到 3 个，但这没有帮助。
我已经提供了网络上的所有屏幕UI，我希望他们能足够清楚地说明问题。有谁知道我在这里做错了什么？
截图：Spark 1 Spark 2 Spark 3 Spark 4 Spark 5 Spark 6
排队和未完成的批处理作业的输出是
callForeachRDD at NativeMethodAccessorImpl.java:0
org.apache.spark.streaming.api.python.PythonDStream.callForeachRDD(PythonDStream.scala)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
py4j.Gateway.invoke(Gateway.java:282)
py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
py4j.commands.CallCommand.execute(CallCommand.java:79)
py4j.GatewayConnection.run(GatewayConnection.java:238)
java.lang.Thread.run(Thread.java:748)
编辑： 我注意到在进程启动时会记录错误。我现在才意识到这一点，因为这个过程不会停止。错误是：
May 11 18:13:04 aida bash[5323]: 20/05/11 18:13:04 ERROR Utils: Uncaught exception in thread stdout writer for python3
May 11 18:13:04 aida bash[5323]: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]: at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]: at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRunner$$anon.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: Exception in thread "stdout writer for python3" java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]: at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]: at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRunner$$anon.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: 20/05/11 18:13:04 ERROR Utils: Uncaught exception in thread stdout writer for python3
May 11 18:13:04 aida bash[5323]: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]: at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]: at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRunner$$anon.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: Exception in thread "stdout writer for python3" java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]: at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]: at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRunner$$anon.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: 20/05/11 18:13:04 ERROR Utils: Uncaught exception in thread stdout writer for python3
May 11 18:13:04 aida bash[5323]: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]: at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]: at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRunner$$anon.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: Exception in thread "stdout writer for python3" java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]: at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$$anon.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]: at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]: at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.PythonRunner$$anon.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]: at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
谁能帮我解决这些错误？
Kafka 与 spark jars 的依赖关系存在冲突... 所以要么不使用 lz 压缩，要么使用 snappy 压缩，它会起作用
或按照答案