无法解析大师 URL:'spark.bluemix.net'
Could not parse Master URL: 'spark.bluemix.net'
我正在尝试在我的台式机上从 RStudio 运行 连接到 Bluemix 上的 IBM Spark 即服务 运行。
我已经从自动配置的 RStudio 环境 运行 复制了 config.yml
在 IBM 的 Data Science Experience 上:
default:
method: "shell"
CS-DSX:
method: "bluemix"
spark.master: "spark.bluemix.net"
spark.instance.id: "myinstanceid"
tenant.id: "mytenantid"
tenant.secret: "mytenantsecret"
hsui.url: "https://cdsx.ng.bluemix.net"
我正在尝试这样连接:
install.packages("sparklyr")
library(sparklyr)
spark_install(version = "1.6.2") # installed spark to '~/Library/Caches/spark/spark-1.6.2-bin-hadoop2.6'
spark_home = '~/Library/Caches/spark/spark-1.6.2-bin-hadoop2.6'
config = spark_config(file = "./config.yml", use_default = FALSE, config = "CSX-DSX")
sc <- spark_connect(spark_home = spark_home, config = config)
错误:
17/03/07 09:36:19 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Could not parse Master URL: 'spark.bluemix.net'
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2735)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:522)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2281)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
关于 Whosebug 的其他一些问题也有类似的错误消息,但他们没有尝试连接到 Bluemix 上的 Spark 服务 运行。
更新 1
我已经将 config.yml 更改为如下所示:
default:
method: "bluemix"
spark.master: "spark://spark.bluemix.net:7070"
spark.instance.id: "7a4089bf-3594-4fdf-8dd1-7e9fd7607be5"
tenant.id: "sdd1-7e9fd7607be53e-39ca506ba762"
tenant.secret: "6146a713-949f-4d4e-84c3-9913d2165b9e"
hsui.url: "https://cdsx.ng.bluemix.net"
...我的连接代码如下所示:
install.packages("sparklyr")
library(sparklyr)
spark_install(version = "1.6.2")
spark_home = '~/Library/Caches/spark/spark-1.6.2-bin-hadoop2.6'
config = spark_config(file = "./config.yml", use_default = FALSE)
sc <- spark_connect(spark_home = spark_home, config = config)
但是,现在的错误是:
Error in force(code) :
Failed during initialize_connection: java.lang.NullPointerException
at org.apache.spark.SparkContext.<init>(SparkContext.scala:583)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2281)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sparklyr.Invoke$.invoke(invoke.scala:94)
...
库试图解析一个 URL,但你给它一个主机名。
尝试 spark://spark.bluemix.net
spark.master
。
请关注博客 post http://datascience.ibm.com/blog/access-ibm-analytics-for-apache-spark-from-rstudio/ 从 DSX RStudio 连接 Bluemix SparkaaS。
我收到了工程团队的以下回复:
RStudio desktop version doesn't support at this time to use sparklyr package to connect Bluemix SparkaaS service
我正在尝试在我的台式机上从 RStudio 运行 连接到 Bluemix 上的 IBM Spark 即服务 运行。
我已经从自动配置的 RStudio 环境 运行 复制了 config.yml
在 IBM 的 Data Science Experience 上:
default:
method: "shell"
CS-DSX:
method: "bluemix"
spark.master: "spark.bluemix.net"
spark.instance.id: "myinstanceid"
tenant.id: "mytenantid"
tenant.secret: "mytenantsecret"
hsui.url: "https://cdsx.ng.bluemix.net"
我正在尝试这样连接:
install.packages("sparklyr")
library(sparklyr)
spark_install(version = "1.6.2") # installed spark to '~/Library/Caches/spark/spark-1.6.2-bin-hadoop2.6'
spark_home = '~/Library/Caches/spark/spark-1.6.2-bin-hadoop2.6'
config = spark_config(file = "./config.yml", use_default = FALSE, config = "CSX-DSX")
sc <- spark_connect(spark_home = spark_home, config = config)
错误:
17/03/07 09:36:19 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Could not parse Master URL: 'spark.bluemix.net'
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2735)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:522)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2281)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
关于 Whosebug 的其他一些问题也有类似的错误消息,但他们没有尝试连接到 Bluemix 上的 Spark 服务 运行。
更新 1
我已经将 config.yml 更改为如下所示:
default:
method: "bluemix"
spark.master: "spark://spark.bluemix.net:7070"
spark.instance.id: "7a4089bf-3594-4fdf-8dd1-7e9fd7607be5"
tenant.id: "sdd1-7e9fd7607be53e-39ca506ba762"
tenant.secret: "6146a713-949f-4d4e-84c3-9913d2165b9e"
hsui.url: "https://cdsx.ng.bluemix.net"
...我的连接代码如下所示:
install.packages("sparklyr")
library(sparklyr)
spark_install(version = "1.6.2")
spark_home = '~/Library/Caches/spark/spark-1.6.2-bin-hadoop2.6'
config = spark_config(file = "./config.yml", use_default = FALSE)
sc <- spark_connect(spark_home = spark_home, config = config)
但是,现在的错误是:
Error in force(code) :
Failed during initialize_connection: java.lang.NullPointerException
at org.apache.spark.SparkContext.<init>(SparkContext.scala:583)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2281)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sparklyr.Invoke$.invoke(invoke.scala:94)
...
库试图解析一个 URL,但你给它一个主机名。
尝试 spark://spark.bluemix.net
spark.master
。
请关注博客 post http://datascience.ibm.com/blog/access-ibm-analytics-for-apache-spark-from-rstudio/ 从 DSX RStudio 连接 Bluemix SparkaaS。
我收到了工程团队的以下回复:
RStudio desktop version doesn't support at this time to use sparklyr package to connect Bluemix SparkaaS service