线程 "main" java.lang.IllegalStateException 中的异常：在没有活动 SparkEnv 的情况下无法使用 'spark' 方案检索文件

Question

我是 spark 和 cassandra 的新手，从 github 获得了一个示例并尝试运行下面的应用程序 link

生成 jar 文件后，尝试使用以下语法执行

C:\Users\user\Desktop\softwares\spark-2.4.3-bin-hadoop2.7\spark-2.4.3-bin-hadoop2.7\bin>spark-submit --class com.github.boneill42.JavaDemo --master spark://localhost:7077
C:\Users\user\git\spark-on-cassandra-quickstart\target/spark-on-cassandra-0.0.1-SNAPSHOT-jar-with-dependencies.jar spark://localhost:7077 localhost

以下是我面临的问题

19/06/08 22:59:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.IllegalStateException: Cannot retrieve files with 'spark' scheme without an active SparkEnv.
        at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:690)
        at org.apache.spark.deploy.DependencyUtils$.downloadFile(DependencyUtils.scala:137)
        at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment.apply(SparkSubmit.scala:367)
        at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment.apply(SparkSubmit.scala:367)
        at scala.Option.map(Option.scala:146)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:366)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

请帮我解决问题

Answer 1

在你的情况下，你似乎想在 standalone mode

开始

spark://HOST:PORT   Connect to the given Spark standalone cluster master.
The port must be whichever one your master is configured to use, which is 7077 by default.

你先启动spark master和worker吗？

启动大师

./sbin/start-master.sh

发射工人

./bin/spark-class org.apache.spark.deploy.worker.Worker  spark://localhost:7077 -c 1 -m 512M

启动master和worker之后，就可以重新提交作业了。

线程 "main" java.lang.IllegalStateException 中的异常：在没有活动 SparkEnv 的情况下无法使用 'spark' 方案检索文件

Exception in thread "main" java.lang.IllegalStateException: Cannot retrieve files with 'spark' scheme without an active SparkEnv

windows

apache-spark

spark-cassandra-connector

cassandra-3.0