Sparkexception：写入行时任务失败（Kubernetes 上的 Spark）

Question

我在 Kubernetes（Azure Kubernetes 服务）上有 Apache Spark 2.4.1 环境。

Spark容器镜像由官方二进制文件(spark-2.4.1-bin-hadoop2.7.tgz)制作而成。它在示例程序（例如 PI 计算）上运行良好。

但我使用我的 Scala 程序，该程序使用 MlLib 并保存 Word2Vec 模型，Spark returns 出现以下错误：

19/04/21 09:08:00 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 29, 10.244.0.43, executor 1): org.apache.spark.SparkException: Task failed while writing rows.
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write.apply(FileFormatWriter.scala:170)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write.apply(FileFormatWriter.scala:169)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun.apply(Executor.scala:403)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:409)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.7-c798b2d2-1676-4e8a-bc38-a0d90c37c80d-libsnappyjava.so: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /tmp/snappy-1.1.7-c798b2d2-1676-4e8a-bc38-a0d90c37c80d-libsnappyjava.so)
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:179)
    at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:154)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
    at org.apache.parquet.hadoop.codec.SnappyCompressor.compress(SnappyCompressor.java:67)
    at org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:81)
    at org.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:92)
    at org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.compress(CodecFactory.java:165)
    at org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:95)
    at org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:147)
    at org.apache.parquet.column.impl.ColumnWriterV1.flush(ColumnWriterV1.java:235)
    at org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:122)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:172)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:114)
    at org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:42)
    at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:57)
    at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:74)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask.apply(FileFormatWriter.scala:247)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask.apply(FileFormatWriter.scala:242)
    at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
    ... 10 more

你有什么建议吗？

Answer 1

根据错误消息指出 *libsnappyjava.so 找不到 ld-linux-x86-64.so.2。这是一个 glibc 动态加载器。所以你有两个解决方案：

使用其他压缩库，例如 gzip。
编辑您的 DockerFile 在您的 docker 映像中安装 libc6-compat

参考：

Answer 2

添加以下运行句创建 Spark 容器的 Dockerfile 后问题得到解决。

RUN ln -s /lib/libc.musl-x86_64.so.1 /lib/ld-linux-x86-64.so.2

Sparkexception：写入行时任务失败（Kubernetes 上的 Spark）

Spark Exception : Task failed while writing rows (Spark on Kuberenetes)

apache-spark

kubernetes

apache-spark-mllib

azure-aks