如何让 groovysh 与 apache spark 一起工作

Question

我成功地将 Apache Spark 与 Groovy 一起使用，但是我没有运气使用 groovysh 作为交互式 spark shell。

Groovy Shell (2.5.0-beta-3, JVM: 1.8.0_161)
Type ':help' or ':h' for help.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
groovy:000> :grab org.apache.spark:spark-sql_2.11:2.2.1
groovy:000> import org.apache.spark.sql.*
===> org.apache.spark.sql.*
groovy:000> spark = SparkSession.builder().master("local[*]").getOrCreate()
===> org.apache.spark.sql.SparkSession@14201a90
groovy:000> test = spark.read().csv('test.csv')
ERROR java.lang.LinkageError:
loader constraint violation: when resolving method "java.lang.management.ManagementFactory.newPlatformMXBeanProxy(Ljavax/management/MBeanServerConnection;Ljava/lang/String;Ljava/lang/Class;)Ljava/lang/Object;" the class loader (instance of org/codehaus/groovy/tools/RootLoader) of the current class, org/apache/spark/util/SizeEstimator$, and the class loader (instance of <bootloader>) for the method's defining class, java/lang/management/ManagementFactory, have different Class objects for the type javax/management/MBeanServerConnection used in the signature
        at org.apache.spark.util.SizeEstimator$.getIsCompressedOops (SizeEstimator.scala:149)
        at org.apache.spark.util.SizeEstimator$.initialize (SizeEstimator.scala:112)
        at org.apache.spark.util.SizeEstimator$.<init> (SizeEstimator.scala:105)
        at org.apache.spark.util.SizeEstimator$.<clinit> (SizeEstimator.scala)
        at org.apache.spark.sql.execution.datasources.SharedInMemoryCache$$anon.weigh (FileStatusCache.scala:109)
        at org.apache.spark.sql.execution.datasources.SharedInMemoryCache$$anon.weigh (FileStatusCache.scala:107)
        at org.spark_project.guava.cache.LocalCache$Segment.setValue (LocalCache.java:2222)
        at org.spark_project.guava.cache.LocalCache$Segment.put (LocalCache.java:2944)
        at org.spark_project.guava.cache.LocalCache.put (LocalCache.java:4212)
        at org.spark_project.guava.cache.LocalCache$LocalManualCache.put (LocalCache.java:4804)
        at org.apache.spark.sql.execution.datasources.SharedInMemoryCache$$anon.putLeafFiles (FileStatusCache.scala:152)
        at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$listLeafFiles.apply (InMemoryFileIndex.scala:128)
        at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$listLeafFiles.apply (InMemoryFileIndex.scala:126)
        at scala.collection.mutable.ResizableArray$class.foreach (ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach (ArrayBuffer.scala:48)
        at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.listLeafFiles (InMemoryFileIndex.scala:126)
        at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.refresh0 (InMemoryFileIndex.scala:90)
        at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.<init> (InMemoryFileIndex.scala:66)
        at org.apache.spark.sql.execution.datasources.DataSource.tempFileIndex$lzycompute (DataSource.scala:129)
        at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$tempFileIndex (DataSource.scala:120)
        at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema (DataSource.scala:134)
        at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation (DataSource.scala:353)
        at org.apache.spark.sql.DataFrameReader.load (DataFrameReader.scala:178)
        at org.apache.spark.sql.DataFrameReader.csv (DataFrameReader.scala:533)
        at org.apache.spark.sql.DataFrameReader.csv (DataFrameReader.scala:412)
        at org.apache.spark.sql.DataFrameReader$csv.call (Unknown Source)

另一方面，看似等效的 groovy 脚本工作得很好

@Grab('org.apache.spark:spark-sql_2.11:2.2.1')
import org.apache.spark.sql.*

def spark = SparkSession.builder().master("local[*]").getOrCreate()

def test = spark.read().csv("test.csv")
test.show()

我正在寻找一种方法来修复上述错误并了解 groovysh 环境与常规 groovy 脚本执行有何不同。

Answer 1

幸运的是新的Spark 2.3.0版本不再有这个问题：

Groovy Shell (2.5.0-beta-3, JVM: 1.8.0_161)
Type ':help' or ':h' for help.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
groovy:000> :grab org.apache.spark:spark-sql_2.11:2.3.0
groovy:000> import org.apache.spark.sql.*
===> org.apache.spark.sql.*
groovy:000> spark = SparkSession.builder().master("local[*]").getOrCreate()
===> org.apache.spark.sql.SparkSession@1de85972
groovy:000> test = spark.read().csv('test.csv')
===> [_c0: string, _c1: string ... 1 more field]
groovy:000> test.show()
+---+---+---+
|_c0|_c1|_c2|
+---+---+---+
|  1|  2|  3|
+---+---+---+

===> null

如何让 groovysh 与 apache spark 一起工作

how to make groovysh work with apache spark

groovy

groovyshell

apache-spark

groovysh