datastax

Question

dsetool 状态

DC: dc1     Workload: Cassandra       Graph: no
======================================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--   Address          Load             Owns                 VNodes                                               Rack         Health [0,1]
UN   192.168.1.130     810.47 MiB       ?                    256                                              2a           0.90
UN   192.168.1.131     683.53 MiB       ?                    256                                          2a           0.90
UN   192.168.1.132      821.33 MiB       ?                    256                                          2a           0.90

DC: dc2     Workload: Analytics       Graph: no     Analytics Master: 192.168.2.131
    =========================================================================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--   Address          Load             Owns                 VNodes                                           Rack         Health [0,1]
UN   192.168.2.130     667.05 MiB       ?                    256                                          2a           0.90
UN   192.168.2.131     845.48 MiB       ?                    256                                          2a           0.90
UN   192.168.2.132       887.92 MiB       ?                    256                                          2a           0.90

当我尝试启动 spark-submit 作业时

dse -u user -p password spark-submit  --class com.sparkLauncher  test.jar prf

我收到以下错误（已编辑）

ERROR 2017-09-14 20:14:14,174 org.apache.spark.deploy.rm.DseAppClient$ClientEndpoint: Failed to connect to DSE resource manager
java.io.IOException: Failed to register with master: dse://?

.....

Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: The method DseResourceManager.registerApplication does not exist. Make sure that the required component for that method is active/enabled

.....

ERROR 2017-09-14 20:14:14,179 org.apache.spark.deploy.rm.DseSchedulerBackend: Application has been killed. Reason: Failed to connect to DSE resource manager: Failed to register with master: dse://?
org.apache.spark.SparkException: Exiting due to error from cluster scheduler: Failed to connect to DSE resource manager: Failed to register with master: dse://?

.....

WARN  2017-09-14 20:14:14,179 org.apache.spark.deploy.rm.DseSchedulerBackend: Application ID is not initialized yet.
ERROR 2017-09-14 20:14:14,384 org.apache.spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem

ERROR 2017-09-14 20:14:14,387 org.apache.spark.deploy.DseSparkSubmitBootstrapper: Failed to start or submit Spark application
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem

我可以确认我已经授予了本文档中提到的权限，https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/security/secAuthSpark.html 我正在 AWS 上尝试这个，如果这有所不同，我可以确认节点之间的路由都是开放的。我可以从任何 spark 节点启动 spark shell，可以调出 Spark UI，可以从 cqlsh 命令获得 spark master

任何指点都会有所帮助，在此先感谢！

Answer 1

主地址必须指向启用分析的有效数据中心中的一个或多个节点。

Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: 
The method DseResourceManager.registerApplication does not exist. 
Make sure that the required component for that method is active/enabled```

表示连接的节点未启用分析。

如果您运行来自非分析节点，您仍然必须指向主 ui 中的分析节点之一。

dse://[Spark node address[:port number]]?[parameter name=parameter value;]...

默认情况下，dse://? url 连接到本地主机作为初始集群连接。

有关详细信息，请参阅 documentation。

Answer 2

出于某种原因我无法确定，我可以运行它在集群模式下提到但在客户端模式下不能

datastax - 无法连接到 spark-submit 上的 DSE 资源管理器

datastax - Failed to connect to DSE resource manager on spark-submit

cassandra

datastax-enterprise

apache-spark