Scala 和 Spark：createStream 时出错

Question

错误：使用 Scala 和 Twitter4j 在 Spark 中创建推文流。下面是我的代码片段：

scala> val ssc = new StreamingContext(sc, Seconds(10))
scala> val cb = new ConfigurationBuilder
scala>cb.setDebugEnabled(true).setOAuthConsumerKey("**********").setOAuthConsume
scala> val auth = new OAuthAuthorization(cb.build)
scala> val tweets = TwitterUtils.createStream(ssc,auth)

error: overloaded method value createStream with alternatives: (jssc: org.apache.spaark.streaming.api.java.JavaStreamingContext, twitterAuth: twitter4j.auth.Authorization) org.apache.spark.streaming.api.java.JavaReceiverInputDStream[twitter4j.Status] (jssc:org.apache.spark.streaming.api.java.JavaStreamingContext, filters: Array[String])org.apache.spark.streaming.api.java.JavaReceiverInputDStream[twitter4j.Status] (ssc:org.apache.spark.streaming.StreamingContext,twitterAuth: Option[twitter4j.auth.Authorization],filters: Seq[String],storageLevel: org.apache.spark.storage.StorageLevel)org.apache.spark.streaming.dstream.ReceiverInputDStream[twitter4j.Status] cannot be applied to (org.apache.spark.streaming.StreamingContext, twitter4j.auth.OAuthAuthorization)

Answer 1

您使用的重载 expects a StreamingContext and an Option[Authorization]，而不是 Authorization。

这应该有效：

val tweets = TwitterUtils.createStream(ssc, Some(auth))

Scala 和 Spark：createStream 时出错

Scala and Spark: Error while createStream

scala

spark-streaming