Spark 2.1.0：读取压缩的 csv 文件

Question

我正在尝试将压缩的 csv 文件 (.bz2) 读取为 DataFrame。我的代码如下

// read the data
    Dataset<Row> rData = spark.read().option("header", true).csv(input);

这在我尝试 IDE 时有效。我可以读取数据并处理它，但是当我尝试使用 Maven 构建它并在命令行上运行时，我收到以下错误

    Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: csv. Please find packages at http://spark.apache.org/third-party-projects.html
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:569)
    at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86)
    at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:325)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
    at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:415)
    at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:352)
    at com.cs6240.Driver.main(Driver.java:28)
Caused by: java.lang.ClassNotFoundException: csv.DefaultSource
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$$anonfun$apply.apply(DataSource.scala:554)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$$anonfun$apply.apply(DataSource.scala:554)
    at scala.util.Try$.apply(Try.scala:192)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun.apply(DataSource.scala:554)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun.apply(DataSource.scala:554)
    at scala.util.Try.orElse(Try.scala:84)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:554)
    ... 7 more

我不确定我是否遗漏了什么。读取 csv 文件是否有一些依赖性？根据文档，Spark 2.x.x 内置了对此的支持。

Answer 1

我按照此答案中的步骤解决了这个问题。

组装插件基本上有一些问题，当我使用阴影插件并使用这个

<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>

Spark 2.1.0：读取压缩的 csv 文件

Spark 2.1.0 : Reading compressed csv file

java

apache-spark

spark-dataframe

apache-spark-mllib