roc_auc 在 VotingClassifier 中，在 scikit-learn (sklearn) 中使用 RandomForestClassifier

Question

我正在尝试为我构建的硬投票分类器计算 roc_auc。我用可重现的例子展示了代码。现在我想计算 roc_auc 分数并绘制 ROC 曲线，但不幸的是我收到以下错误 predict_proba is not available when voting='hard'

# Voting Ensemble for Classification
import pandas
from sklearn import datasets
from sklearn import model_selection
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.ensemble import VotingClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import MultinomialNB
from sklearn.cross_validation import StratifiedShuffleSplit
from sklearn.model_selection import cross_val_predict
from sklearn.model_selection import cross_val_score
from sklearn.metrics import make_scorer,confusion_matrix, f1_score, precision_score, recall_score, cohen_kappa_score,accuracy_score,roc_curve
import numpy as np
np.random.seed(42)
iris = datasets.load_iris()
X = iris.data[:, :4]  # we only take the first two features.
Y = iris.target
print(Y)
seed = 7
kfold = model_selection.KFold(n_splits=10, random_state=seed)
# create the sub models
estimators = []
model1 = LogisticRegression()
estimators.append(('logistic', model1))
model2 = RandomForestClassifier(n_estimators=200, max_depth=3, random_state=0)
estimators.append(('RandomForest', model2))
model3 = MultinomialNB()
estimators.append(('NaiveBayes', model3))
model4=SVC(probability=True)
estimators.append(('svm', model4))
model5=DecisionTreeClassifier()
estimators.append(('Cart', model5))
# create the ensemble model
print('Majority Class Labels (Majority/Hard Voting)')
ensemble = VotingClassifier(estimators,voting='hard')
#accuracy
results = model_selection.cross_val_score(ensemble, X, Y, cv=kfold,scoring='accuracy')
y_pred = cross_val_predict(ensemble, X ,Y, cv=10)
print("Accuracy ensemble model : %0.2f (+/- %0.2f) " % (results.mean(), results.std() ))
print(results.mean())
#recall
recall_scorer = make_scorer(recall_score, pos_label=1)
recall = cross_val_score(ensemble, X, Y, cv=kfold, scoring=recall_scorer)
print('Recall', np.mean(recall), recall)
# Precision
precision_scorer = make_scorer(precision_score, pos_label=1)
precision = cross_val_score(ensemble, X, Y, cv=kfold, scoring=precision_scorer)
print('Precision', np.mean(precision), precision)
#f1_score
f1_scorer = make_scorer(f1_score, pos_label=1)
f1_score = cross_val_score(ensemble, X, Y, cv=kfold, scoring=f1_scorer)
print('f1_score ', np.mean(f1_score ),f1_score )
#roc_auc_score
roc_auc_score = cross_val_score(ensemble, X, Y, cv=kfold, scoring='roc_auc')
print('roc_auc_score ', np.mean(roc_auc_score ),roc_auc_score )

Answer 1

要计算roc_auc指标，您首先需要

替换：ensemble = VotingClassifier(estimators,voting='hard')

与: ensemble = VotingClassifier(estimators,voting='soft').

Next，最后两行代码将抛出一个错误:

roc_auc_score = cross_val_score(ensemble, X, Y, cv=3, scoring='roc_auc')
print('roc_auc_score ', np.mean(roc_auc_score ),roc_auc_score )

ValueError: multiclass format is not supported

这是正常的，因为在 Y 你有 3 classes (np.unique(Y) == array([0, 1, 2])).

您不能将 roc_auc 用作多个 class 模型的单个汇总指标。如果你愿意，你可以计算 **per-class roc_auc.**

如何解决：

1) 只使用两个classes来计算roc_auc_score

2) 在调用之前提前使用标签二值化roc_auc_score

roc_auc 在 VotingClassifier 中，在 scikit-learn (sklearn) 中使用 RandomForestClassifier

roc_auc in VotingClassifier, RandomForestClassifier in scikit-learn (sklearn)

python

decision-tree

roc

scikit-learn

ensemble-learning