如何在 Python 中使用 Keras 的 scikit-learn 评估指标函数?
How to employ the scikit-learn evaluation metrics functions with Keras in Python?
Keras 提供了定义自定义评估指标的可能性——我对 F 指标的变体很感兴趣,例如F1、F2 等由 scikit learn 提供——但指示我们通过调用在这方面受到限制的 Keras 后端函数来执行此操作。
我的目标是将这些指标与 Keras 的早期停止方法结合使用。所以我应该找到一种方法将指标与 Keras 模型的学习过程相结合。 (当然,在 learning/fitting 过程之外,我可以简单地用结果调用 Scikit-Learn)。
我有哪些选择?
更新
使用来自 Kaggle 的 titanic_all_numeric 数据集实施 Aaron 的解决方案后,我得到以下信息:
# Compile the model
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy', f1])
# Fit the model
hist = model.fit(predictors, target, validation_split = 0.3)
Train on 623 samples, validate on 268 samples
Epoch 1/1
623/623 [==============================] - 0s 642us/step - loss: 0.8037 - acc: 0.6132 - f1: 0.6132 - val_loss: 0.5815 - val_acc: 0.7537 - val_f1: 0.7537
# Compile the model
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fit the model
hist = model.fit(predictors, target, validation_split = 0.3)
Train on 623 samples, validate on 268 samples
Epoch 1/1
623/623 [==============================] - 0s 658us/step - loss: 0.8148 - acc: 0.6404 - val_loss: 0.7056 - val_acc: 0.7313
# Compile the model
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = [f1])
# Fit the model
hist = model.fit(predictors, target, validation_split = 0.3)
Train on 623 samples, validate on 268 samples
Epoch 1/1
623/623 [==============================] - 0s 690us/step - loss: 0.6554 - f1: 0.6709 - val_loss: 0.5107 - val_f1: 0.7612
我想知道这些结果是否合适。一次,准确率和f1分数是一样的。
您可以将您的预测和标签从您的 keras 模型传递到任何 scikit-learn 函数以进行评估。例如,如果您正在处理分类问题,您可以利用 scikit-learn 中的 classification_report
,它提供诸如精度、召回率、f1 分数等指标,例如(示例代码直接取自他们的文档):
from sklearn.metrics import classification_report
y_true = [0, 1, 2, 2, 2]
y_pred = [0, 0, 2, 2, 1]
target_names = ['class 0', 'class 1', 'class 2']
print(classification_report(y_true, y_pred, target_names=target_names))
precision recall f1-score support
class 0 0.50 1.00 0.67 1
class 1 0.00 0.00 0.00 1
class 2 1.00 0.67 0.80 3
micro avg 0.60 0.60 0.60 5
macro avg 0.50 0.56 0.49 5
weighted avg 0.70 0.60 0.61 5
更新:
如果您想将指标纳入 keras 培训中,请使用:
from keras import backend as K
def f1(y_true, y_pred):
def recall(y_true, y_pred):
"""Recall metric.
Only computes a batch-wise average of recall.
Computes the recall, a metric for multi-label classification of
how many relevant items are selected.
"""
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
recall = true_positives / (possible_positives + K.epsilon())
return recall
def precision(y_true, y_pred):
"""Precision metric.
Only computes a batch-wise average of precision.
Computes the precision, a metric for multi-label classification of
how many selected items are relevant.
"""
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
precision = true_positives / (predicted_positives + K.epsilon())
return precision
precision = precision(y_true, y_pred)
recall = recall(y_true, y_pred)
return 2*((precision*recall)/(precision+recall+K.epsilon()))
model.compile(loss='binary_crossentropy',
optimizer= "adam",
metrics=[f1])
Keras 提供了定义自定义评估指标的可能性——我对 F 指标的变体很感兴趣,例如F1、F2 等由 scikit learn 提供——但指示我们通过调用在这方面受到限制的 Keras 后端函数来执行此操作。
我的目标是将这些指标与 Keras 的早期停止方法结合使用。所以我应该找到一种方法将指标与 Keras 模型的学习过程相结合。 (当然,在 learning/fitting 过程之外,我可以简单地用结果调用 Scikit-Learn)。
我有哪些选择?
更新
使用来自 Kaggle 的 titanic_all_numeric 数据集实施 Aaron 的解决方案后,我得到以下信息:
# Compile the model
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy', f1])
# Fit the model
hist = model.fit(predictors, target, validation_split = 0.3)
Train on 623 samples, validate on 268 samples
Epoch 1/1
623/623 [==============================] - 0s 642us/step - loss: 0.8037 - acc: 0.6132 - f1: 0.6132 - val_loss: 0.5815 - val_acc: 0.7537 - val_f1: 0.7537
# Compile the model
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fit the model
hist = model.fit(predictors, target, validation_split = 0.3)
Train on 623 samples, validate on 268 samples
Epoch 1/1
623/623 [==============================] - 0s 658us/step - loss: 0.8148 - acc: 0.6404 - val_loss: 0.7056 - val_acc: 0.7313
# Compile the model
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = [f1])
# Fit the model
hist = model.fit(predictors, target, validation_split = 0.3)
Train on 623 samples, validate on 268 samples
Epoch 1/1
623/623 [==============================] - 0s 690us/step - loss: 0.6554 - f1: 0.6709 - val_loss: 0.5107 - val_f1: 0.7612
我想知道这些结果是否合适。一次,准确率和f1分数是一样的。
您可以将您的预测和标签从您的 keras 模型传递到任何 scikit-learn 函数以进行评估。例如,如果您正在处理分类问题,您可以利用 scikit-learn 中的 classification_report
,它提供诸如精度、召回率、f1 分数等指标,例如(示例代码直接取自他们的文档):
from sklearn.metrics import classification_report
y_true = [0, 1, 2, 2, 2]
y_pred = [0, 0, 2, 2, 1]
target_names = ['class 0', 'class 1', 'class 2']
print(classification_report(y_true, y_pred, target_names=target_names))
precision recall f1-score support
class 0 0.50 1.00 0.67 1
class 1 0.00 0.00 0.00 1
class 2 1.00 0.67 0.80 3
micro avg 0.60 0.60 0.60 5
macro avg 0.50 0.56 0.49 5
weighted avg 0.70 0.60 0.61 5
更新: 如果您想将指标纳入 keras 培训中,请使用:
from keras import backend as K
def f1(y_true, y_pred):
def recall(y_true, y_pred):
"""Recall metric.
Only computes a batch-wise average of recall.
Computes the recall, a metric for multi-label classification of
how many relevant items are selected.
"""
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
recall = true_positives / (possible_positives + K.epsilon())
return recall
def precision(y_true, y_pred):
"""Precision metric.
Only computes a batch-wise average of precision.
Computes the precision, a metric for multi-label classification of
how many selected items are relevant.
"""
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
precision = true_positives / (predicted_positives + K.epsilon())
return precision
precision = precision(y_true, y_pred)
recall = recall(y_true, y_pred)
return 2*((precision*recall)/(precision+recall+K.epsilon()))
model.compile(loss='binary_crossentropy',
optimizer= "adam",
metrics=[f1])