spark-submit '无法将 'startDate' 强制转换为格式化日期（长）'

Question

出现错误：错误：当我运行 spark 提交如下时，无法将 'startDate' 强制为格式化日期（长）：

dse -u cassandra -p cassandra spark-submit --class com.abc.rm.Total_count \
   --master dse://x.x.x.x:9042 TotalCount.jar \
   "2024-06-11 00:00:00.000+0000" "2027-11-15 00:00:00.000+0000" \
   10-118-16-132.bbc.ds.com pramod history

下面是我的代码：

package com.abc.rm
import com.datastax.spark.connector._
import org.apache.spark.SparkContext
object Total_count {
  def main(args: Array[String]):Unit = {
    var startDate = args(0)
    var endDate = args(1)
    val master = args(2)
    var ks = args(3)
    var table_name = args(4)
    println("startDate-->"+startDate)
    println("endDate-->"+endDate)
    println("master-->"+master)

    val conf = new org.apache.spark.SparkConf().setAppName("Total_count")
       .set("spark.cassandra.connection.host", master)
       .set("spark.cassandra.auth.username","cassandra")
       .set("spark.cassandra.auth.password","cassandra")

    var sc = new SparkContext(conf)

    val rdd = sc.cassandraTable("pramod", "history")
        .where("sent_date>='startDate' and sent_date <='endDate'")
        .cassandraCount()
    println("count--> "+rdd)
    sc.stop()
    System.exit(1)
}}

我怎么能pass/convert这个论点。

Answer 1

您没有传递参数，而是按字面意思传递字符串 startDate 和 endDate。要使其正常工作，您需要将其写为

.where(s"sent_date>='$startDate' and sent_date <='$endDate'")

spark-submit '无法将 'startDate' 强制转换为格式化日期（长）'

spark-submit 'Unable to coerce 'startDate' to a formatted date (long)'

scala

datastax

apache-spark

spark-submit