java.lang.NumberFormatException 在 Scala 中用于 mllib

Question

我只是想开发一个简单的 K-Means 算法示例，但在加载和清理数据时遇到很多问题。

这是我的代码：

import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.clustering.KMeans
import org.apache.spark.sql.functions._


val crimeRDD = sc.textFile("/home/borja/spark/pruebas/AlgoritmoClusterizacion/filter_data1.csv")

val header = crimeRDD.first

val data = crimeRDD.filter (justData => justData != header)

//Spark doesn't allow more than 22 element
case class crimeReport (Record_ID: Int, Agency_Name: String, City: String, State: String, Year: Int, Month: String, Crime_Type: String, Crime_Solved: String, Victim_Sex: String, Victim_Age: Int, Victim_Race: String, Perpetrator_Sex: String, Perpetrator_Age: Int, Perpetrator_Race: String, Relationship: String, Victim_Count: Int)

val data_split = data.map(line => line.split(","))

val allData = data_split.map(p => crimeReport(p(0).trim.toInt, p(1).trim.toString, p(2).trim.toString, p(3).trim.toString, p(4).trim.toInt, p(5).trim.toString, p(6).trim.toString, p(7).trim.toString, p(8).trim.toString, p(9).trim.toInt, p(10).trim.toString, p(11).trim.toString, p(12).trim.toInt, p(13).trim.toString,p(14).trim.toString, p(15).trim.toInt))

val allDF = allData.toDF()
allDF.printSchema
//allDF.show(100)

val rowsRDD = allDF.rdd.map(r => (r.getInt(0),r.getString(1),r.getString(2), r.getString(3),r.getInt(4), r.getString(5), r.getString(6), r.getString(7),r.getString(8), r.getInt(9), r.getString(10), r.getString(11),r.getInt(12), r.getString(13),r.getString(14), r.getInt(15)))

rowsRDD.cache()

val features_vector = allDF.rdd.map(r => Vectors.dense(r.getInt(0)))

features_vector.cache()

val KMeansModel = KMeans.train(features_vector,2,40)

但是我遇到了这个错误

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 43.0 failed 1 times, most recent failure: Lost task 0.0 in stage 43.0 (TID 57, localhost, executor driver): java.lang.NumberFormatException: For input string: "Jersey"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)

我不明白，因为我正在用函数 .trim 清理白色和空格，是吗？关键是异常被抛出，因为当我执行 .toInt 时有一些字符串值，对吗？那么我该如何过滤它们，因为有 65k 行。

这里有一组数据：

Record ID,Agency Name,City,State,Year,Month,Crime Type,Crime Solved,Victim Sex,Victim Age,Victim Race,Perpetrator Sex,Perpetrator Age,Perpetrator Race,Relationship,Victim Count
1,Anchorage,Anchorage,Alaska,1980,January,Murder or Manslaughter,Yes,Male,14,Native American/Alaska Native,Male,15,Native American/Alaska Native,Acquaintance,0
13504,Atlantic City,Atlantic,Jersey,1980,January,Murder or Manslaughter,Yes,Male,40,Black,Female,50,Black,Acquaintance,0
13505,Atlantic City,Atlantic,Jersey,1980,January,Murder or Manslaughter,No,Male,23,Black,Unknown,0,Unknown,Unknown,0
13506,Atlantic City,Atlantic,Jersey,1980,January,Murder or Manslaughter,No,Male,52,White,Unknown,0,Unknown,Unknown,0
13507,Atlantic City,Atlantic,Jersey,1980,March,Murder or Manslaughter,Yes,Male,35,Black,Male,23,Black,Unknown,0
13508,Atlantic City,Atlantic,Jersey,1980,March,Murder or Manslaughter,No,Male,25,Black,Unknown,0,Unknown,Unknown,0
13647,Jersey City,Hudson,Jersey,1980,October,Murder or Manslaughter,No,Female,50,White,Unknown,0,Unknown,Unknown,2
13648,Jersey City,Hudson,Jersey,1980,March,Murder or Manslaughter,Yes,Female,60,White,Male,36,White,Father,1
13649,Jersey City,Hudson,Jersey,1980,June,Murder or Manslaughter,Yes,Female,52,Black,Male,26,Black,Unknown,1
13650,Jersey City,Hudson,Jersey,1980,October,Murder or Manslaughter,No,Male,2,White,Unknown,0,Unknown,Unknown,2
13651,Jersey City,Hudson,Jersey,1980,January,Murder or Manslaughter,Yes,Female,68,Black,Male,0,Black,Unknown,0
13652,Jersey City,Hudson,Jersey,1980,January,Murder or Manslaughter,Yes,Female,22,Black,Male,23,Black,Unknown,0
13653,Jersey City,Hudson,Jersey,1980,January,Murder or Manslaughter,Yes,Female,16,White,Male,33,White,Acquaintance,0
13654,Jersey City,Hudson,Jersey,1980,January,Murder or Manslaughter,Yes,Male,34,White,Male,18,White,Acquaintance,0
13655,Jersey City,Hudson,Jersey,1980,February,Murder or Manslaughter,No,Male,29,Black,Unknown,0,Unknown,Unknown,0
13656,Jersey City,Hudson,Jersey,1980,February,Murder or Manslaughter,No,Male,42,White,Unknown,0,Unknown,Unknown,0

Answer 1

是的，当无法将 object 转换为 integer 值时会抛出错误消息。为什么不试试 Option 或 Try
例如

Option(p(0).trim.toInt) getOrElse 0

或者

Try(p(0).trim.toInt) getOrElse 0

这应该足以避免此类转换错误

并且您在代码中提到 case class 仅支持 22 个元素。此限制已在较新版本的 spark 中删除。希望回答对你有帮助

已编辑：
注意： Option 只捕捉到 NullPointerException，因此不会解决转换异常。 Try 处理几乎所有类型的 Exception，因此 Try 应该是转换异常的选项。

Answer 2

您可以使用Try来处理异常。您可以在 scala 中使用旧的 Java 样式 try/catch 或 Try 的功能方式。下面是在 scala 中处理异常的功能方法。

Try(p(0).trim.toInt) match {
    case Success(result) => result  
    case Failure (ex) => {
      ex.printStackTrace()
      //return a default value as you want i have returned 0
      0
    }
  }

如果语句通过则尝试returns成功，如果语句失败或出现异常则returns失败。我希望这更容易理解，也更实用。

java.lang.NumberFormatException 在 Scala 中用于 mllib

java.lang.NumberFormatException in Scala for mllib

scala

k-means

apache-spark

apache-spark-mllib