使用 Scala TTL 选项将 Spark 保存到 Cassandra

Question

我在 Scala 中使用 Cassandra Spark 连接器。这是我的示例代码，用于将数据保存到 Cassandra。

val data = rdd.map
(f => new CassandraRow(IndexedSeq("pk", "count"), IndexedSeq(f._1, f._2.toString())))

data.saveToCassandra("keyspace", "table")

然后，我尝试使用TTL。这是我添加 TTL 的示例代码。

添加导入

import com.datastax.spark.connector.writer.{TTLOption, WriteConf}

并将 TTL 添加到 saveToCassandra

data.saveToCassandra
("keyspace", "table", writeConf = WriteConf(ttl = TTLOption.constant(604800))

但是，当我编译它时，出现错误。

错误信息

[error] bad symbolic reference. A signature in TTLOption.class refers to term streaming
[error] in package org.apache.spark which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling TTLOption.class.
[error] one error found
[error] (compile:compile) Compilation failed
[error] Total time: 4 s, completed Mar 31, 2015 11:55:14 AM

build.sbt

name := "Cassandra"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.2.1"

libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.2.1"

libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.10" % "1.2.0-rc2"

我的代码有问题吗？？

Answer 1

这是当前连接器中的一个已知错误，与对流式 spark 库。

https://datastax-oss.atlassian.net/browse/SPARKC-113

如果这是阻塞的并且您需要解决方法，您可以将 spark-streaming 库作为依赖项包含在构建文件中。

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.2.1"

使用 Scala TTL 选项将 Spark 保存到 Cassandra

Spark saving to Cassandra with Scala TTL Option

scala

cassandra

apache-spark