从 PySpark ParamGrid 中提取 MLP 层

Question

所以我在 Pipeline 和 CrossValidator 之后从 PySpark 模型中提取超参数时遇到了问题。

我在 Whosebug 上找到了以下答案：

这非常有帮助，下面一行对我有用：

modelOnly.bestModel.stages[-1]._java_obj.parent().getRegParam()

新问题是我是运行 MLP，在尝试提取图层时，我得到一个随机字符串而不是 Python 列表之类的东西。

结果：

StepSize: 0.03

Layers: [I@db98c25

我的代码大致是：

trainer = MultilayerPerceptronClassifier(featuresCol='features', 
                                     labelCol='label', 
                                     predictionCol='prediction', 
                                     maxIter=100, 
                                     tol=1e-06, 
                                     seed=1331, 
                                     layers=layers1, 
                                     blockSize=128, 
                                     stepSize=0.03, 
                                     solver='l-bfgs', 
                                     initialWeights=None, 
                                     probabilityCol='probability', 
                                     rawPredictionCol='rawPrediction')

pipeline = Pipeline(stages=[assembler1,stringIdx,trainer])

paramGrid = ParamGridBuilder() \
.addGrid(trainer.maxIter, [10]) \
.addGrid(trainer.tol, [1e-06]) \
.addGrid(trainer.stepSize, [0.03]) \
.addGrid(trainer.layers, [layers2]) \
.build()

crossval = CrossValidator(estimator=pipeline,
                      estimatorParamMaps=paramGrid,
                      evaluator=MulticlassClassificationEvaluator(metricName="accuracy"),
                      numFolds=3)

cvModel = crossval.fit(df)

mybestmodel = cvModel.bestModel

java_model = mybestmodel.stages[-1]._java_obj

print("StepSize: ", end='')
print(java_model.parent().getStepSize())
print("Layers: ", end='')
print(java_model.parent().getLayers())

我是运行 Spark 2.3.2.

我错过了什么？

谢谢:)

Answer 1

那不是随机字符串，而是 representation of the corresponding Java object。

虽然理论上你可以

[x for x in mybestmodel.stages[-1]._java_obj.parent().getLayers()]

there is really no need for that

layers

array of layer sizes including input and output layers.

New in version 1.6.0.

即

mybestmodel.stages[-1].layers

从 PySpark ParamGrid 中提取 MLP 层

Extracting MLP Layers from PySpark ParamGrid

apache-spark

pyspark

apache-spark-ml