Cannot run ALS.train, error: java.lang.IllegalArgumentException
Cannot run ALS.train, error: java.lang.IllegalArgumentException
我正在尝试 运行 ALS of PySpark。我复制并粘贴了 link 中提供的示例代码。但是，错误 java.lang.IllegalArgumentException 出现在行：
model = ALS.train(ratings, rank, numIterations)
请问这里需要排查哪些可能的问题？
我的 Spark 版本是 2.2.1，我的 Java 版本是 9.0.4。但是，我不确定 Spark 是否使用了正确的 Java 版本，尽管我设置了环境路径并且命令 "java -version" 执行 return "9.0.4".
错误：
Py4JJavaError: An error occurred while calling
z:org.apache.spark.mllib.api.python.SerDe.pythonToJava. :
java.lang.IllegalArgumentException at
org.apache.xbean.asm5.ClassReader.(Unknown Source) at
org.apache.xbean.asm5.ClassReader.(Unknown Source) at
org.apache.xbean.asm5.ClassReader.(Unknown Source) at
org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:46)
at
org.apache.spark.util.FieldAccessFinder$$anon$$anonfun$visitMethodInsn.apply(ClosureCleaner.scala:443)
at
org.apache.spark.util.FieldAccessFinder$$anon$$anonfun$visitMethodInsn.apply(ClosureCleaner.scala:426)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach.apply(TraversableLike.scala:733)
at
scala.collection.mutable.HashMap$$anon$$anonfun$foreach.apply(HashMap.scala:103)
at
scala.collection.mutable.HashMap$$anon$$anonfun$foreach.apply(HashMap.scala:103)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at
scala.collection.mutable.HashMap$$anon.foreach(HashMap.scala:103) at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.util.FieldAccessFinder$$anon.visitMethodInsn(ClosureCleaner.scala:426)
at org.apache.xbean.asm5.ClassReader.a(Unknown Source) at
org.apache.xbean.asm5.ClassReader.b(Unknown Source) at
org.apache.xbean.asm5.ClassReader.accept(Unknown Source) at
org.apache.xbean.asm5.ClassReader.accept(Unknown Source) at
org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean.apply(ClosureCleaner.scala:257)
at
org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean.apply(ClosureCleaner.scala:256)
at scala.collection.immutable.List.foreach(List.scala:381) at
org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:256)
at
org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:156)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2294) at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions.apply(RDD.scala:794)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions.apply(RDD.scala:793)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) at
org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:793) at
org.apache.spark.mllib.api.python.SerDeBase.pythonToJava(PythonMLLibAPI.scala:1349)
at
org.apache.spark.mllib.api.python.SerDe.pythonToJava(PythonMLLibAPI.scala)
at jdk.internal.reflect.GeneratedMethodAccessor76.invoke(Unknown
Source) at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564) at
py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at
py4j.Gateway.invoke(Gateway.java:280) at
py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79) at
py4j.GatewayConnection.run(GatewayConnection.java:214) at
java.base/java.lang.Thread.run(Thread.java:844)
我找到问题了。 Spark 2.2.1 不支持 Java 9.0.4。也许我误解了教程的"Java 8+"部分。
如果有人遇到和我一样的错误，就去Java 1.8.0！
我正在尝试 运行 ALS of PySpark。我复制并粘贴了 link 中提供的示例代码。但是，错误 java.lang.IllegalArgumentException 出现在行：
model = ALS.train(ratings, rank, numIterations)
请问这里需要排查哪些可能的问题？
我的 Spark 版本是 2.2.1，我的 Java 版本是 9.0.4。但是，我不确定 Spark 是否使用了正确的 Java 版本，尽管我设置了环境路径并且命令 "java -version" 执行 return "9.0.4".
错误：
Py4JJavaError: An error occurred while calling z:org.apache.spark.mllib.api.python.SerDe.pythonToJava. : java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) at org.apache.xbean.asm5.ClassReader.(Unknown Source) at org.apache.xbean.asm5.ClassReader.(Unknown Source) at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:46) at org.apache.spark.util.FieldAccessFinder$$anon$$anonfun$visitMethodInsn.apply(ClosureCleaner.scala:443) at org.apache.spark.util.FieldAccessFinder$$anon$$anonfun$visitMethodInsn.apply(ClosureCleaner.scala:426) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach.apply(TraversableLike.scala:733) at scala.collection.mutable.HashMap$$anon$$anonfun$foreach.apply(HashMap.scala:103) at scala.collection.mutable.HashMap$$anon$$anonfun$foreach.apply(HashMap.scala:103) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at scala.collection.mutable.HashMap$$anon.foreach(HashMap.scala:103) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) at org.apache.spark.util.FieldAccessFinder$$anon.visitMethodInsn(ClosureCleaner.scala:426) at org.apache.xbean.asm5.ClassReader.a(Unknown Source) at org.apache.xbean.asm5.ClassReader.b(Unknown Source) at org.apache.xbean.asm5.ClassReader.accept(Unknown Source) at org.apache.xbean.asm5.ClassReader.accept(Unknown Source) at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean.apply(ClosureCleaner.scala:257) at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean.apply(ClosureCleaner.scala:256) at scala.collection.immutable.List.foreach(List.scala:381) at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:256) at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:156) at org.apache.spark.SparkContext.clean(SparkContext.scala:2294) at org.apache.spark.rdd.RDD$$anonfun$mapPartitions.apply(RDD.scala:794) at org.apache.spark.rdd.RDD$$anonfun$mapPartitions.apply(RDD.scala:793) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:793) at org.apache.spark.mllib.api.python.SerDeBase.pythonToJava(PythonMLLibAPI.scala:1349) at org.apache.spark.mllib.api.python.SerDe.pythonToJava(PythonMLLibAPI.scala) at jdk.internal.reflect.GeneratedMethodAccessor76.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.base/java.lang.Thread.run(Thread.java:844)
我找到问题了。 Spark 2.2.1 不支持 Java 9.0.4。也许我误解了教程的"Java 8+"部分。
如果有人遇到和我一样的错误，就去Java 1.8.0！