尝试使用 BayesSearchCV 调整 MLPClassifier hidden_layer_sizes 时出错

Question

当尝试调整 sklearn MLPClassifier hidden_layer_sizes 超参数时，使用 BayesSearchCV，我得到一个错误：ValueError: can only convert an array of size 1 to a Python scalar.

但是，当我使用 GridSearchCV 时，效果很好！ 我错过了什么？

这是一个可重现的例子：

from skopt import BayesSearchCV
from skopt.space import Real, Categorical, Integer
from sklearn.datasets import load_iris
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV

X, y = load_iris(True)
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                 train_size=0.75,
                                                 random_state=0)
# this does not work!
opt_bs = BayesSearchCV(MLPClassifier(), 
                     {'learning_rate_init': Real(0.001, 0.05),
                        'solver': Categorical(["adam", 'sgd']), 
                        'hidden_layer_sizes': Categorical([(10,5), (15,10,5)])}, 
                     n_iter=32,
                     random_state=0)

# this one does :)
opt_gs = GridSearchCV(MLPClassifier(), 
                   {'learning_rate_init': [0.001, 0.05],
                        'solver': ["adam", 'sgd'], 
                        'hidden_layer_sizes': [(10,5), (15,10,5)]})
                   
# executes optimization using opt_gs or opt_bs
opt = opt_bs
res = opt.fit(X_train, y_train)
opt

生产：

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-64-78e6d29cae99> in <module>()
     27 # executes optimization using opt_gs or opt_bs
     28 opt = opt_bs
---> 29 res = opt.fit(X_train, y_train)
     30 opt

/usr/local/lib/python3.6/dist-packages/skopt/searchcv.py in fit(self, X, y, groups, callback)
    678                 optim_result = self._step(
    679                     X, y, search_space, optimizer,
--> 680                     groups=groups, n_points=n_points_adjusted
    681                 )
    682                 n_iter -= n_points

/usr/local/lib/python3.6/dist-packages/skopt/searchcv.py in _step(self, X, y, search_space, optimizer, groups, n_points)
    553 
    554         # convert parameters to python native types
--> 555         params = [[np.array(v).item() for v in p] for p in params]
    556 
    557         # make lists into dictionaries

/usr/local/lib/python3.6/dist-packages/skopt/searchcv.py in <listcomp>(.0)
    553 
    554         # convert parameters to python native types
--> 555         params = [[np.array(v).item() for v in p] for p in params]
    556 
    557         # make lists into dictionaries

/usr/local/lib/python3.6/dist-packages/skopt/searchcv.py in <listcomp>(.0)
    553 
    554         # convert parameters to python native types
--> 555         params = [[np.array(v).item() for v in p] for p in params]
    556 
    557         # make lists into dictionaries

ValueError: can only convert an array of size 1 to a Python scalar

Answer 1

不幸的是，BayesSearchCV 只接受分类、整数或实数类型值的参数。在您的情况下，没有问题 w.r.t learning_rate_init 和 solver 参数，因为它们分别明确定义为 Real 和 Categorical，问题出在 hidden_layer_sizes 其中您已将神经元的数量声明为 Categorical 值，在这种情况下是元组，并且 BayesSearchCV 尚未具备处理元组中搜索空间的能力，请参阅 here 了解更多详细信息.但是，作为临时 hack，您可以围绕 MLPClassifier 创建自己的包装器，以正确识别估算器的参数。请参考以下代码片段获取示例：

from skopt import BayesSearchCV
from skopt.space import Integer
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPRegressor, MLPClassifier
from sklearn.base import BaseEstimator, ClassifierMixin
import itertools

X, y = load_iris(True)
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                 train_size=0.75,
                                                 random_state=0)

class MLPWrapper(BaseEstimator, ClassifierMixin):
    def __init__(self, layer1=10, layer2=10, layer3=10):
        self.layer1 = layer1
        self.layer2 = layer2
        self.layer3 = layer3

    def fit(self, X, y):
        model = MLPClassifier(
            hidden_layer_sizes=[self.layer1, self.layer2, self.layer3]
        )
        model.fit(X, y)
        self.model = model
        return self

    def predict(self, X):
        return self.model.predict(X)

    def score(self, X, y):
        return self.model.score(X, y)


opt = BayesSearchCV(
    estimator=MLPWrapper(),
    search_spaces={
        'layer1': Integer(10, 100),
        'layer2': Integer(10, 100),
        'layer3': Integer(10, 100)
    },
    n_iter=11
)

opt.fit(X_train, y_train)
opt.score(X_test,y_test)
0.9736842105263158

注意：这假设您构建了一个具有三层的 MLP 网络。您可以根据需要修改它。此外，创建一个 class 来构造具有任意层数的任何 MLP 变得有点棘手。

尝试使用 BayesSearchCV 调整 MLPClassifier hidden_layer_sizes 时出错

Error when trying to tune MLPClassifier hidden_layer_sizes using BayesSearchCV

python

scikit-learn

mlp

bayessearchcv