winutils spark windows 安装 env_variable

Question

我正在尝试在 windows 10 上安装 Spark 1.6.1，到目前为止我已经完成了以下操作...

下载spark 1.6.1，解压到某个目录，然后设置SPARK_HOME
下载scala 2.11.8，解压到某个目录然后设置SCALA_HOME
设置 _JAVA_OPTION 环境变量
通过仅下载 zip 目录然后设置 HADOOP_HOME env 变量从 https://github.com/steveloughran/winutils.git 下载 winutils。（不确定这是否不正确，因为权限被拒绝，我无法克隆该目录）。

当我回家时运行 bin\spark-shell 我得到

'C:\Program' is not recognized as an internal or external command, operable program or batch file.

我一定是遗漏了一些东西，我不明白我怎么能在 windows 环境中运行安装 bash 脚本。但希望我不需要理解就可以让它工作。我一直在关注这个人的教程 - https://hernandezpaul.wordpress.com/2016/01/24/apache-spark-installation-on-windows-10/ 。任何帮助，将不胜感激。

Answer 1

您需要下载 winutils 可执行文件，而不是源代码。

可以下载here, or if you really want the entire Hadoop distribution you can find the 2.6.0 binaries here。然后，您需要将 HADOOP_HOME 设置为包含 winutils.exe 的目录。

此外，确保放置 Spark 的目录是一个不包含空格的目录，这一点非常重要，否则将无法运行。

设置完成后，您不会启动 spark-shell.sh，您会启动 spark-shell.cmd:

C:\Spark\bin>spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties
To adjust logging level use sc.setLogLevel("INFO")
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.1
      /_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_91)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
16/05/18 19:31:56 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-core-3.2.10.jar."
16/05/18 19:31:56 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-api-jdo-3.2.6.jar."
16/05/18 19:31:56 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-rdbms-3.2.9.jar."
16/05/18 19:31:56 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/05/18 19:31:56 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/05/18 19:32:01 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/05/18 19:32:01 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/05/18 19:32:07 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-core-3.2.10.jar."
16/05/18 19:32:07 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-api-jdo-3.2.6.jar."
16/05/18 19:32:07 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-rdbms-3.2.9.jar."
16/05/18 19:32:07 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/05/18 19:32:08 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/05/18 19:32:12 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/05/18 19:32:12 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
SQL context available as sqlContext.

scala>

Answer 2

在 windows 您需要明确指定 hadoop 二进制文件的位置。

以下是设置 spark-scala 独立应用程序的步骤。

下载 winutil.exe 并将其放置到某些 folder/directory 下的 bin 文件夹中，比如 c:\hadoop\bin

完整路径类似于 c:\hadoop\bin\winutil.exe

现在在创建 sparkSession 时我们需要指定此路径。参考下面的代码片段：

套餐com.test.config

 import org.apache.spark.sql.SparkSession

 object Spark2Config extends Serializable{
            System.setProperty("hadoop.home.dir", "C:\hadoop")
            val spark = SparkSession.builder().appName("app_name").master("local").getOrCreate()       
 }

winutils spark windows 安装 env_variable

winutils spark windows installation env_variable

windows

git

scala

apache-spark

apache-spark-standalone