'Input validation failed' Spark MLlib 多 class 逻辑回归中的错误

Question

我有一个包含 5000 行和 401 列的训练集，其中第 1 列是标签，其余 400 列是特征。我正在尝试使用 pyspark mllib 进行多类逻辑回归。请在下面找到我的代码。我必须承认，这不是优化/编写良好的代码，因为我仍然是 python/pyspark.

领域的新手

tset=sio.loadmat('ex3data1.mat') # load the training set from a mat file
X=tset['X']                      # read the X,y values
y=tset['y']
print(X.shape) # works!
print(y.shape)
sp= 
SparkSession.builder.master("local").appName("multiclassifier").getOrCreate()
sc=sp.sparkContext
XY=np.concatenate((y,X),axis=1) # 5000x401 where the first col is the label.
print(XY[0:2])

上面打印的示例输出。请注意我只打印第一行

[[  1.00000000e+01   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
8.56059680e-06   1.94035948e-06  -7.37438725e-04  -8.13403799e-03
-1.86104473e-02  -1.87412865e-02  -1.87572508e-02  -1.90963542e-03
-1.64039011e-02  -3.78191381e-03   3.30347316e-04   1.27655229e-05
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   1.16421569e-04
    1.20052179e-04  -1.40444581e-02  -2.84542484e-02   8.03826593e-02
    2.66540339e-01   2.73853746e-01   2.78729541e-01   2.74293607e-01
    2.24676403e-01   2.77562977e-02  -7.06315478e-03   2.34715414e-04
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   1.28335523e-17  -3.26286765e-04
   -1.38651604e-02   8.15651552e-02   3.82800381e-01   8.57849775e-01
    1.00109761e+00   9.69710638e-01   9.30928598e-01   1.00383757e+00
    9.64157356e-01   4.49256553e-01  -5.60408259e-03  -3.78319036e-03
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    5.10620915e-06   4.36410675e-04  -3.95509940e-03  -2.68537241e-02
    1.00755014e-01   6.42031710e-01   1.03136838e+00   8.50968614e-01
    5.43122379e-01   3.42599738e-01   2.68918777e-01   6.68374643e-01
    1.01256958e+00   9.03795598e-01   1.04481574e-01  -1.66424973e-02
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    2.59875260e-05  -3.10606987e-03   7.52456076e-03   1.77539831e-01
    7.92890120e-01   9.65626503e-01   4.63166079e-01   6.91720680e-02
   -3.64100526e-03  -4.12180405e-02  -5.01900656e-02   1.56102907e-01
    9.01762651e-01   1.04748346e+00   1.51055252e-01  -2.16044665e-02
    0.00000000e+00   0.00000000e+00   0.00000000e+00   5.87012352e-05
   -6.40931373e-04  -3.23305249e-02   2.78203465e-01   9.36720163e-01
    1.04320956e+00   5.98003217e-01  -3.59409041e-03  -2.16751770e-02
   -4.81021923e-03   6.16566793e-05  -1.23773318e-02   1.55477482e-01
    9.14867477e-01   9.20401348e-01   1.09173902e-01  -1.71058007e-02
    0.00000000e+00   0.00000000e+00   1.56250000e-04  -4.27724104e-04
   -2.51466503e-02   1.30532561e-01   7.81664862e-01   1.02836583e+00
    7.57137601e-01   2.84667194e-01   4.86865128e-03  -3.18688725e-03
    0.00000000e+00   8.36492601e-04  -3.70751123e-02   4.52644165e-01
    1.03180133e+00   5.39028101e-01  -2.43742611e-03  -4.80290033e-03
    0.00000000e+00   0.00000000e+00  -7.03635621e-04  -1.27262443e-02
    1.61706648e-01   7.79865383e-01   1.03676705e+00   8.04490400e-01
    1.60586724e-01  -1.38173339e-02   2.14879493e-03  -2.12622549e-04
    2.04248366e-04  -6.85907627e-03   4.31712963e-04   7.20680947e-01
    8.48136063e-01   1.51383408e-01  -2.28404366e-02   1.98971950e-04
    0.00000000e+00   0.00000000e+00  -9.40410539e-03   3.74520505e-02
    6.94389110e-01   1.02844844e+00   1.01648066e+00   8.80488426e-01
    3.92123945e-01  -1.74122413e-02  -1.20098039e-04   5.55215142e-05
   -2.23907271e-03  -2.76068376e-02   3.68645493e-01   9.36411169e-01
    4.59006723e-01  -4.24701797e-02   1.17356610e-03   1.88929739e-05
    0.00000000e+00   0.00000000e+00  -1.93511951e-02   1.29999794e-01
    9.79821705e-01   9.41862388e-01   7.75147704e-01   8.73632241e-01
    2.12778350e-01  -1.72353349e-02   0.00000000e+00   1.09937426e-03
   -2.61793751e-02   1.22872879e-01   8.30812662e-01   7.26501773e-01
    5.24441863e-02  -6.18971913e-03   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00  -9.36563862e-03   3.68349741e-02
    6.99079299e-01   1.00293583e+00   6.05704402e-01   3.27299224e-01
   -3.22099249e-02  -4.83053002e-02  -4.34069138e-02  -5.75151144e-02
    9.55674190e-02   7.26512627e-01   6.95366966e-01   1.47114481e-01
   -1.20048679e-02  -3.02798203e-04   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00  -6.76572712e-04  -6.51415556e-03
    1.17339359e-01   4.21948410e-01   9.93210937e-01   8.82013974e-01
    7.45758734e-01   7.23874268e-01   7.23341725e-01   7.20020340e-01
    8.45324959e-01   8.31859739e-01   6.88831870e-02  -2.77765012e-02
    3.59136710e-04   7.14869281e-05   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   1.53186275e-04   3.17353553e-04
   -2.29167177e-02  -4.14402914e-03   3.87038450e-01   5.04583435e-01
    7.74885876e-01   9.90037446e-01   1.00769478e+00   1.00851440e+00
    7.37905042e-01   2.15455291e-01  -2.69624864e-02   1.32506127e-03
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    2.36366422e-04  -2.26031454e-03  -2.51994485e-02  -3.73889910e-02
    6.62121228e-02   2.91134498e-01   3.23055726e-01   3.06260315e-01
    8.76070942e-02  -2.50581917e-02   2.37438725e-04   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   6.20939216e-18   6.72618320e-04
   -1.13151411e-02  -3.54641066e-02  -3.88214912e-02  -3.71077412e-02
   -1.33524928e-02   9.90964718e-04   4.89176960e-05   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00]]

end of print output.

pXYdf=pd.DataFrame(XY)
sXYdf=sp.createDataFrame(pXYdf)

from pyspark.mllib.classification import LogisticRegressionWithLBFGS, 
LogisticRegressionModel
import pyspark.mllib.regression as reg

trainingData = sXYdf.rdd.map(lambda x: reg.LabeledPoint(x[0],x[1:]))
trainingData.take(2) # this works!!

以 LabeledPoint 格式输出 1 条记录：（我无法在此处正确设置格式，因为这里有 400 个特征`。

[LabeledPoint(10.0,[0.0,0.0,0.0,0.0,0.0,0.0,....,8.56059679589e-06, 1.94035947712e06,.........]),

lrm=LogisticRegressionWithLBFGS.train(trainingData,numClasses=10)

然而，这似乎不起作用，因为我收到了 py4javaError。不幸的是，我遇到了一些我正在尝试调试的本地 pyspark 执行问题，因此无法剪切和粘贴错误消息。

Answer 1

我猜你遇到了错误

: org.apache.spark.SparkException: Input validation failed.

Spark MLlib 逻辑回归中有一个-实际上未记录- 的特征，根据该特征对于 k classes，标签应该是 0, 1, ..., k-1，即它们必须从零开始。无论你在相关的 Pyspark 文档中盯着看多久，你都不会找到这个要求，只是因为它不存在 - 要看到它，你必须挖掘相关的 Scala source code，其中提到：

* @note Labels used in Logistic Regression should be {0, 1, ..., k - 1}

从您提供的代码片段来看，您似乎有 10 个 class，标记为 1, 2, ..., 10；这将不起作用，并且会产生上述错误。这是一个带有虚拟数据和 3 classes:

的简短演示

sc.version
# u'2.1.1'

from pyspark.mllib.classification import LogisticRegressionWithLBFGS, LogisticRegressionModel
from pyspark.mllib.regression import LabeledPoint

data = [LabeledPoint(3.0, [4.6,3.6,1.0,0.2]), # 3 classes, labeled 1, 2, 3
        LabeledPoint(3.0, [5.7,4.4,1.5,0.4]),
        LabeledPoint(1.0, [6.7,3.1,4.4,1.4]),
        LabeledPoint(2.0, [4.8,3.4,1.6,0.2]),
        LabeledPoint(1.0, [4.4,3.2,1.3,0.2])]

model = LogisticRegressionWithLBFGS.train(sc.parallelize(data), numClasses=3)
[...]
Py4JJavaError: An error occurred while calling o173.trainLogisticRegressionModelWithLBFGS. 
: org.apache.spark.SparkException: Input validation failed.

因此，如果确实是这种情况，您需要从标签中减去 1，从而将它们从 1, 2, ..., 10 转换为 0, 1, ..., 9。这是在我的 3-class 虚拟数据中执行此操作的快速方法：

new_data = sc.parallelize(data).map(lambda x: LabeledPoint(x.label-1, x.features))
new_data.collect() # for demonstration purposes only
# [LabeledPoint(2.0, [4.6,3.6,1.0,0.2]), # 3 classes, labeled 0, 1, 2
#  LabeledPoint(2.0, [5.7,4.4,1.5,0.4]),
#  LabeledPoint(0.0, [6.7,3.1,4.4,1.4]),
#  LabeledPoint(1.0, [4.8,3.4,1.6,0.2]),
#  LabeledPoint(0.0, [4.4,3.2,1.3,0.2])]

model = LogisticRegressionWithLBFGS.train(new_data, numClasses=3) # works OK

我在 blog post 中就此 Input validation fail 和其他未记录的 and/or 反直觉的内容进行了详细讨论 blog post，您可能会发现它们很有用。

'Input validation failed' Spark MLlib 多 class 逻辑回归中的错误

'Input validation failed' error in Spark MLlib multi-class logistic regression

machine-learning

logistic-regression

apache-spark

pyspark

apache-spark-mllib