在 sklearn 中超参数调整后查找模型的准确度、精确度和召回率

Question

我有一个二元分类问题，为此我选择了 3 种算法，Logistic 回归、SVM 和 Adaboost。我对每一个都使用网格搜索和 k 折交叉验证来找到最佳超参数集。现在，我需要根据准确性、精确度和召回率来选择最佳模型。但问题是我无法找到任何合适的方法来检索这些信息。我的代码如下：

from sklearn.model_selection import GridSearchCV
from sklearn.metrics.scorer import make_scorer
from sklearn import cross_validation

# TODO: Initialize the classifier
clfr_A = LogisticRegression(random_state=128)
clfr_B = SVC(random_state=128)
clfr_C = AdaBoostClassifier(random_state=128)

lr_param_grid = {'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000] }
svc_param_grid = {'C': [0.001, 0.01, 0.1, 1, 10], 'gamma' : [0.001, 0.01, 0.1, 1]}
adb_param_grid = {'n_estimators' : [50,100,150,200,250,500],'learning_rate' : [.5,.75,1.0,1.25,1.5,1.75,2.0]}

# TODO: Make an fbeta_score scoring object using make_scorer()
scorer = make_scorer(fbeta_score, beta = 0.5)

# TODO: Perform grid search on the classifier using 'scorer' as the scoring method using GridSearchCV()
clfrs = [clfr_A, clfr_B, clfr_C]
params = [lr_param_grid, svc_param_grid, adb_param_grid]

for clfr, param in zip(clfrs, params):
    grid_obj = GridSearchCV(clfr, param, cv=3, scoring=scorer, refit=True)
    grid_fit = grid_obj.fit(features_raw, target_raw)
    print grid_fit.best_estimator_
    print grid_fit.cv_results_

问题是 cv_results_ 给出了很多信息，但除了 mean_test_score，我找不到任何相关信息。此外，我在那里看不到任何与准确性、精确度或召回率相关的指标。

我能想到一种方法来实现它。我可以将 for 循环更改为如下所示：

score_params = ["accuracy", "precision", "recall"]
for clfr, param in zip(clfrs, params):
    grid_obj = GridSearchCV(clfr, param, cv=3, scoring=scorer, refit=True)
    grid_fit = grid_obj.fit(features_raw, target_raw)
    best_clf = grid_fit.best_estimator_
    for score in score_params:
        print score,
        print " : ",
        print cross_val_score(best_clf, features_raw, target_raw, scoring=score, cv=3).mean()

但是有没有更好的方法呢？看来我正在为每个模型多次执行操作。感谢任何帮助。

Answer 1

GridSearchCV 正在执行您提供的操作。您将 f_beta 作为记分员，因此 mean_test_score 将为每个参数组合 f_beta 结果 f_beta。如果您想访问其他指标，您需要明确告诉 GridSearchCV 这样做。

较新版本的 scikit-learn 中的 GridSearchCV 支持多指标评分。所以你可以通过多种类型的得分手。作为 per documentation:

scoring : string, callable, list/tuple, dict or None, default: None

... ...

For evaluating multiple metrics, either give a list of (unique) strings or a dict with names as keys and callables as values.

在此处查看此示例：

http://scikit-learn.org/stable/auto_examples/model_selection/plot_multi_metric_evaluation.html#running-gridsearchcv-using-multiple-evaluation-metrics

并将您的 scoring 参数更改为：

scoring = {'Accuracy': 'accuracy', 
           'FBeta': make_scorer(fbeta_score, beta = 0.5),
           # ... Add others here as you want.
           }

但是现在当你这样做的时候，你还需要改变 refit 参数。由于此处不同的指标会为参数组合给出不同类型的分数，因此您需要在重新拟合估算器时决定 select 哪个。因此，从 refit

的评分字典中选择一个键

for clfr, param in zip(clfrs, params):
    grid_obj = GridSearchCV(clfr, param, cv=3, scoring=scorer, refit='FBeta')
    ...
    ...

在 sklearn 中超参数调整后查找模型的准确度、精确度和召回率

Finding accuracy, precision and recall of a model after hyperparameter tuning in sklearn

python

scikit-learn

cross-validation