为什么 spark 总是尝试连接到 localhost:9042？

Question

当我构建上下文时，我使用了以下参数：

spark.cassandra.connection.host=somehosts&spark.cassandra.auth.username=app&spark.cassandra.auth.password=app

更多详情如下：

{{tstr}}contexts/cassandra?num-cpu-cores=4&memory-per-node=5g&context-factory=spark.jobserver.context.SessionContextFactory&spark.executor.instances=10&spark.scheduler.mode=FAIR&spark.executor.cores=5&spark.cassandra.connection.host=some.host&spark.cassandra.auth.username=app&spark.cassandra.auth.password=app

但是当我开始工作时，出现如下异常：

java.io.IOException: Failed to open native connection to Cassandra at {127.0.0.1}:9042
    at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:168)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun.apply(CassandraConnector.scala:154)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun.apply(CassandraConnector.scala:154)
    at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32)
    at com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69)
    at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:57)
    at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:79)
    at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:111)
    at com.datastax.spark.connector.rdd.partitioner.dht.TokenFactory$.forSystemLocalPartitioner(TokenFactory.scala:98)
    at org.apache.spark.sql.cassandra.CassandraSourceRelation$.apply(CassandraSourceRelation.scala:272)
    at org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.scala:56)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:309)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146)
    at spark.jobserver.SparkSessionJob$class.runJob(SparkSessionJob.scala:16)
    at spark.jobserver.JobManagerActor$$anonfun$getJobFuture.apply(JobManagerActor.scala:594)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1(Future.scala:24)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.TransportException: [localhost/127.0.0.1:9042] Cannot connect))
    at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:233)
    at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:79)
    at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1483)
    at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:399)
    at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:161)
    ... 24 more

我添加了一些日志如下：

log.info(spark.sparkContext.getConf.get("spark.cassandra.connection.host"))
    log.info(spark.sparkContext.getConf.get("spark.cassandra.auth.username"))
    log.info(spark.sparkContext.getConf.get("spark.cassandra.auth.password"))

得到的结果为：

2019-01-18 21:22:57 INFO  root:39 - localhost
2019-01-18 21:22:57 INFO  root:40 - app
2019-01-18 21:22:57 INFO  root:41 - app

似乎其他设置有效只是 spark.cassandra.connection.host 无效。

有人知道怎么解决吗？

Answer 1

现在知道怎么回事了，我错误地将请求正文设置如下：

spark.context-settings {
    spark.cores.max = 10
    passthrough {
      spark.cassandra.connection.host="localhost"
      spark.cassandra.auth.username=app
      spark.cassandra.auth.password=app
    }
}

在代码中，作业服务器将使用 passthrough 来重置 sparkConfig。

为什么 spark 总是尝试连接到 localhost:9042？

why spark always try to connect to localhost:9042?

scala

cassandra

spark-jobserver