Scikit-learn 和 Yellowbrick 给出不同的分数
Scikit-learn and Yellowbrick giving different scores
我正在使用 sklearn 计算分类器的平均精度和 roc_auc,并使用 yellowbrick 绘制 roc_auc 和精度召回曲线。问题是这些包在两个指标上给出了不同的分数,我不知道哪个是正确的。
使用的代码:
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from yellowbrick.classifier import ROCAUC
from yellowbrick.classifier import PrecisionRecallCurve
from sklearn.datasets import make_classification
from sklearn.metrics import roc_auc_score
from sklearn.metrics import average_precision_score
seed = 42
# provides de data
X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0,
n_informative=2, random_state=seed)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
clf_lr = LogisticRegression(random_state=seed)
clf_lr.fit(X_train, y_train)
y_pred = clf_lr.predict(X_test)
roc_auc = roc_auc_score(y_test, y_pred)
avg_precision = average_precision_score(y_test, y_pred)
print(f"ROC_AUC: {roc_auc}")
print(f"Average_precision: {avg_precision}")
print('='*20)
# visualizations
viz3 = ROCAUC(LogisticRegression(random_state=seed))
viz3.fit(X_train, y_train)
viz3.score(X_test, y_test)
viz3.show()
viz4 = PrecisionRecallCurve(LogisticRegression(random_state=seed))
viz4.fit(X_train, y_train)
viz4.score(X_test, y_test)
viz4.show()
代码产生以下输出:
如上所示,指标根据包给出不同的值。在 print 语句中是 scikit-learn 计算的值,而在图中出现注释的是 yellowbrick 计算的值。
由于您使用 scikit-learn 的 predict
方法,您的预测 y_pred
是硬 class 成员资格,而不是概率:
np.unique(y_pred)
# array([0, 1])
但是对于ROC和Precision-Recall计算,应该不是;您传递给这些方法的预测应该是概率,而不是硬 classes。来自 average_precision_score
docs:
y_score: array, shape = [n_samples] or [n_samples, n_classes]
Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as
returned by “decision_function” on some classifiers).
其中 non-thresholded 表示 不难 classes。 roc_auc_score
(docs).
的情况类似
使用以下代码更正此问题,使 scikit-learn 结果与 Yellowbrick return 编辑的结果相同:
y_pred = clf_lr.predict_proba(X_test) # get probabilities
y_prob = np.array([x[1] for x in y_pred]) # keep the prob for the positive class 1
roc_auc = roc_auc_score(y_test, y_prob)
avg_precision = average_precision_score(y_test, y_prob)
print(f"ROC_AUC: {roc_auc}")
print(f"Average_precision: {avg_precision}")
结果:
ROC_AUC: 0.9545954595459546
Average_precision: 0.9541994473779806
由于 Yellowbrick 在内部(并且透明地)处理所有这些计算细节,因此它不会遭受此处制作的手动 scikit-learn 过程中的错误。
请注意,在二进制情况下(如此处),您可以(并且应该)使用 binary=True
参数使您的绘图不那么混乱:
viz3 = ROCAUC(LogisticRegression(random_state=seed), binary=True) # similarly for the PrecisionRecall curve
而且,与人们的直觉预期相反,至少对于二进制情况,ROCAUC
的 score
方法 不会 return 的 AUC,但准确度在 docs:
中指定
viz3.score(X_test, y_test)
# 0.88
# verify this is the accuracy:
from sklearn.metrics import accuracy_score
accuracy_score(y_test, clf_lr.predict(X_test))
# 0.88
我正在使用 sklearn 计算分类器的平均精度和 roc_auc,并使用 yellowbrick 绘制 roc_auc 和精度召回曲线。问题是这些包在两个指标上给出了不同的分数,我不知道哪个是正确的。
使用的代码:
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from yellowbrick.classifier import ROCAUC
from yellowbrick.classifier import PrecisionRecallCurve
from sklearn.datasets import make_classification
from sklearn.metrics import roc_auc_score
from sklearn.metrics import average_precision_score
seed = 42
# provides de data
X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0,
n_informative=2, random_state=seed)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
clf_lr = LogisticRegression(random_state=seed)
clf_lr.fit(X_train, y_train)
y_pred = clf_lr.predict(X_test)
roc_auc = roc_auc_score(y_test, y_pred)
avg_precision = average_precision_score(y_test, y_pred)
print(f"ROC_AUC: {roc_auc}")
print(f"Average_precision: {avg_precision}")
print('='*20)
# visualizations
viz3 = ROCAUC(LogisticRegression(random_state=seed))
viz3.fit(X_train, y_train)
viz3.score(X_test, y_test)
viz3.show()
viz4 = PrecisionRecallCurve(LogisticRegression(random_state=seed))
viz4.fit(X_train, y_train)
viz4.score(X_test, y_test)
viz4.show()
代码产生以下输出:
如上所示,指标根据包给出不同的值。在 print 语句中是 scikit-learn 计算的值,而在图中出现注释的是 yellowbrick 计算的值。
由于您使用 scikit-learn 的 predict
方法,您的预测 y_pred
是硬 class 成员资格,而不是概率:
np.unique(y_pred)
# array([0, 1])
但是对于ROC和Precision-Recall计算,应该不是;您传递给这些方法的预测应该是概率,而不是硬 classes。来自 average_precision_score
docs:
y_score: array, shape = [n_samples] or [n_samples, n_classes]
Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).
其中 non-thresholded 表示 不难 classes。 roc_auc_score
(docs).
使用以下代码更正此问题,使 scikit-learn 结果与 Yellowbrick return 编辑的结果相同:
y_pred = clf_lr.predict_proba(X_test) # get probabilities
y_prob = np.array([x[1] for x in y_pred]) # keep the prob for the positive class 1
roc_auc = roc_auc_score(y_test, y_prob)
avg_precision = average_precision_score(y_test, y_prob)
print(f"ROC_AUC: {roc_auc}")
print(f"Average_precision: {avg_precision}")
结果:
ROC_AUC: 0.9545954595459546
Average_precision: 0.9541994473779806
由于 Yellowbrick 在内部(并且透明地)处理所有这些计算细节,因此它不会遭受此处制作的手动 scikit-learn 过程中的错误。
请注意,在二进制情况下(如此处),您可以(并且应该)使用 binary=True
参数使您的绘图不那么混乱:
viz3 = ROCAUC(LogisticRegression(random_state=seed), binary=True) # similarly for the PrecisionRecall curve
而且,与人们的直觉预期相反,至少对于二进制情况,ROCAUC
的 score
方法 不会 return 的 AUC,但准确度在 docs:
viz3.score(X_test, y_test)
# 0.88
# verify this is the accuracy:
from sklearn.metrics import accuracy_score
accuracy_score(y_test, clf_lr.predict(X_test))
# 0.88