Sparkling Water - 运行 python 脚本作为 Spark 应用程序

Sparkling Water - run python script as a Spark Application

我在将 Sparkling Water 运行 python 脚本作为 Spark 应用程序时遇到了一些问题。我使用此命令在 Spark 上执行我的脚本:

./bin/spark-submit \

--packages ai.h2o:sparkling-water-core_2.10:1.5.12 \

--py-files $SPARKLING_HOME/py/dist/pySparkling-1.5.12-py2.7.egg $SPARKLING_HOME/Python/test.py

我有这个坠落错误:

py4j.protocol.Py4JError: Trying to call a package.

日志:

> Traceback (most recent call last):
  File "/Users/Documents/sparkling-water-1.5.12/Python/test.py", line 5, in <module>
    hc= H2OContext(sc).start()
  File "/Users/Documents/sparkling-water-1.5.12/py/dist/pySparkling-1.5.12-py2.7.egg/pysparkling/context.py", line 72, in __init__
  File "/Users/Documents/sparkling-water-1.5.12/py/dist/pySparkling-1.5.12-py2.7.egg/pysparkling/context.py", line 96, in _do_init
  File "/Users/Documents/spark-1.5.2-bin-hadoop2.4/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 726, in __getattr__
py4j.protocol.Py4JError: Trying to call a package.
16/04/11 16:58:39 INFO SparkContext: Invoking stop() from shutdown hook
16/04/11 16:58:39 INFO SparkUI: Stopped Spark web UI at http://192.168.181.84:4042
16/04/11 16:58:39 INFO DAGScheduler: Stopping DAGScheduler
16/04/11 16:58:39 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/04/11 16:58:39 INFO MemoryStore: MemoryStore cleared
16/04/11 16:58:39 INFO BlockManager: BlockManager stopped
16/04/11 16:58:39 INFO BlockManagerMaster: BlockManagerMaster stopped
16/04/11 16:58:39 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/04/11 16:58:39 INFO SparkContext: Successfully stopped SparkContext
16/04/11 16:58:39 INFO ShutdownHookManager: Shutdown hook called
16/04/11 16:58:39 INFO ShutdownHookManager: Deleting directory /private/var/fold

我该如何解决这个问题?我完全按照手册中的命令进行操作:https://h2o-release.s3.amazonaws.com/h2o/rel-turan/3/docs-website/h2o-docs/booklets/SparklingWaterVignette.pdf

这实际上是我们在 Sparkling Water 团队中知道的一个严重错误,它已在新版本中与其他修补程序一起修复。该错误已修复 (https://0xdata.atlassian.net/browse/SW-107),新版本应该很快就会发布。

我会及时通知您,并在新版本发布时通知您。

2016 年 4 月 29 日编辑

带有修复程序的新版本已经发布。

对于 spark 1.6 - http://h2o-release.s3.amazonaws.com/sparkling-water/rel-1.6/3/index.html

对于 spark 1.5 - http://h2o-release.s3.amazonaws.com/sparkling-water/rel-1.5/14/index.html

您无需再致电 -packages 即可添加气泡水核心。 pySparkling egg 文件已经包含了所有必要的 Java/Scala 类 它需要的。所以你需要做的就是使用 py-files 选项设置 egg 文件,就这样。