Keras 中基于自定义指标的提前停止和学习率计划
Early stopping and learning rate schedule based on custom metric in Keras
我在 Keras 中有一个对象检测模型，我想根据在验证集上计算的平均精度 (mAP) 来监控我的训练。
我已将代码从 tensorflow-models 移植到我的脚本中，该脚本使用提供的模型和数据运行评估。它不是作为 Keras 指标实现的，而是作为独立的 class:
evaluation = SSDEvaluation(model, data, data_size)
mAP = evaluation.evaluate()
我完全可以接受这样的情况。事实上，我不希望它被计算为训练批次，因为它会减慢训练速度。
我的问题是：如何根据每个纪元后计算的指标重用 ReduceLROnPlateau 和 EarlyStopping 回调？
我不确定 SSDEvaluation 是什么，但如果可以接受任何没有开销的平均精度计算，我建议使用 keras callbacks.
以下方法
您希望 oto 使用两个 callbacls 的主要想法 - EarlyStopping 和 ReduceLROnPlateau - 都作用于纪元结束并监视 loss 或 metric 值。他们从 method
的 logs 参数中得到这个值
def on_epoch_end(self, epoch, logs=None):
"""Called at the end of an epoch.
...
"""
- 将实际的 map 发送到日志值，我们强制此方法和所有从日志中获取精度值的回调使用它。 Callbcaks 从这里选择值（this ine int the code - early stopping, and this 一个用于 Reduce LR）。
因此，我们应该 "fake" 记录这两个回调。我想这不是理想的，但可行的解决方案。
这个 类 从回调继承并计算 map 值，他们还避免重新计算 map 通过共享对象 Hub。
from sklearn.metrics import average_precision_score
import keras
from keras.callbacks import Callback, EarlyStopping, ReduceLROnPlateau
class MAPHub:
def __init__(self):
self.map_value = None
- 它只是共享 map 值的中心。可能会引起一些副作用。你可以尽量避免使用它。
def on_epoch_end(self, epoch, logs):
"""self just a callbcak instance"""
if self.last_metric_for_epoch == epoch:
map_ = self.hub.map_value
else:
prediction = self.model.predict(self._data, verbose=1)
map_ = average_precision_score(self._target, prediction)
self.hub.map_value = map_
self.last_metric_for_epoch = epoch
- 此函数计算并共享 map
class EarlyStoppingByMAP(EarlyStopping):
def __init__(self, data, target, hub, *args, **kwargs):
"""
data, target - values and target for the map calculation
hub - shared object to store _map_ value
*args, **kwargs for the super __init__
"""
# I've set monitor to 'acc' here, because you're interested in metric, not loss
super(EarlyStoppingByMAP, self).__init__(monitor='acc', *args, **kwargs)
self._target = target
self._data = data
self.last_metric_for_epoch = -1
self.hub = hub
def on_epoch_end(self, epoch, logs):
"""
epoch is the number of epoch, logs is a dict logs with 'loss' value
and metric 'acc' values
"""
on_epoch_end(self, epoch, logs)
logs['acc'] = self.hub.map_value # "fake" metric with calculated value
print('Go callback from the {}, logs: \n{}'.format(EarlyStoppingByMAP.__name__, logs))
super(EarlyStoppingByMAP, self).on_epoch_end(epoch, logs) # works as a callback fn
class ReduceLROnPlateauByMAP(ReduceLROnPlateau):
def __init__(self, data, target, hub, *args, **kwargs):
# the same as in previous
# I've set monitor to 'acc' here, because you're interested in metric, not loss
super(ReduceLROnPlateauByMAP, self).__init__(monitor='acc', *args, **kwargs)
self._target = target
self._data = data
self.last_metric_for_epoch = -1
self.hub = hub
def on_epoch_end(self, epoch, logs):
on_epoch_end(self, epoch, logs)
logs['acc'] = self.hub.map_value # "fake" metric with calculated value
print('Go callback from the {}, logs: \n{}'.format(ReduceLROnPlateau.__name__, logs))
super(ReduceLROnPlateauByMAP, self).on_epoch_end(epoch, logs) # works as a callback fn
- 注意 不要在构造函数中使用 monitor 参数！您应该使用 'acc'，参数已设置为正确的值。
一些测试：
from keras.datasets import mnist
from keras.models import Model
from keras.layers import Dense, Input
import numpy as np
(X_tr, y_tr), (X_te, y_te) = mnist.load_data()
X_tr = (X_tr / 255.).reshape((60000, 784))
X_te = (X_te / 255.).reshape((10000, 784))
def binarize_labels(y):
y_bin = np.zeros((len(y), len(np.unique(y))))
y_bin[range(len(y)), y] = 1
return y_bin
y_train_bin, y_test_bin = binarize_labels(y_tr), binarize_labels(y_te)
inp = Input(shape=(784,))
x = Dense(784, activation='relu')(inp)
x = Dense(256, activation='relu')(x)
out = Dense(10, activation='softmax')(x)
model = Model(inp, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')
- 一个简单的 "test suite"。现在去适应它：
hub = MAPHub() # instentiate a hub
# I will use default params except patience as example, set it to 1 and 5
early_stop = EarlyStoppingByMAP(X_te, y_test_bin, hub, patience=1) # Patience is EarlyStopping's param
reduce_lt = ReduceLROnPlateauByMAP(X_te, y_test_bin, hub, patience=5) # Patience is ReduceLR's param
history = model.fit(X_tr, y_train_bin, epochs=10, callbacks=[early_stop, reduce_lt])
Out:
Epoch 1/10
60000/60000 [==============================] - 12s 207us/step - loss: 0.1815
10000/10000 [==============================] - 1s 59us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.18147853660446903, 'acc': 0.9934216252519924}
10000/10000 [==============================] - 0s 40us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.18147853660446903, 'acc': 0.9934216252519924}
Epoch 2/10
60000/60000 [==============================] - 12s 197us/step - loss: 0.0784
10000/10000 [==============================] - 0s 40us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.07844233275586739, 'acc': 0.9962269038764738}
10000/10000 [==============================] - 0s 41us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.07844233275586739, 'acc': 0.9962269038764738}
Epoch 3/10
60000/60000 [==============================] - 12s 197us/step - loss: 0.0556
10000/10000 [==============================] - 0s 40us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.05562876497630107, 'acc': 0.9972085346550085}
10000/10000 [==============================] - 0s 40us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.05562876497630107, 'acc': 0.9972085346550085}
Epoch 4/10
60000/60000 [==============================] - 12s 198us/step - loss: 0.0389
10000/10000 [==============================] - 0s 41us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.0388911374788188, 'acc': 0.9972696414934574}
10000/10000 [==============================] - 0s 41us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.0388911374788188, 'acc': 0.9972696414934574}
Epoch 5/10
60000/60000 [==============================] - 12s 197us/step - loss: 0.0330
10000/10000 [==============================] - 0s 39us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.03298293751536124, 'acc': 0.9959456176387349}
10000/10000 [==============================] - 0s 39us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.03298293751536124, 'acc': 0.9959456176387349}
好的，看起来至少可以提前停止。我想，ReduceLROnPlateau 因为他们使用相同的日志和相似的逻辑 - 如果设置了适当的参数。
如果你不想使用sklearn函数，但是SSDEvaluation（我就是找不到它是什么）——你可以很容易地采用on_epoch_method函数来处理这个评估功能。
希望对您有所帮助。
您可以使用 LambdaCallback 来更新您的 logs 对象：
假设你的 evaluation.evaluate() returns 像 {'val/mAP': value} 这样的字典，你可以这样做：
eval_callback = LambdaCallback(
on_epoch_end=lambda epoch, logs: logs.update(evaluation.evaluate())
)
这里的技巧是 logs 将进一步传递给其他回调，因此它们可以直接访问值：
early_stopping = EarlyStopping(monitor='val/mAP', min_delta=0.0, patience=10, verbose=1, mode='max')
它将自动出现在 CSVLogger 和任何其他回调中。但请注意 eval_callback 必须在使用回调列表中的值的任何回调之前：
callbacks = [eval_callback, early_stopping]
我在 Keras 中有一个对象检测模型，我想根据在验证集上计算的平均精度 (mAP) 来监控我的训练。
我已将代码从 tensorflow-models 移植到我的脚本中，该脚本使用提供的模型和数据运行评估。它不是作为 Keras 指标实现的，而是作为独立的 class:
evaluation = SSDEvaluation(model, data, data_size)
mAP = evaluation.evaluate()
我完全可以接受这样的情况。事实上，我不希望它被计算为训练批次，因为它会减慢训练速度。
我的问题是：如何根据每个纪元后计算的指标重用 ReduceLROnPlateau 和 EarlyStopping 回调？
我不确定 SSDEvaluation 是什么，但如果可以接受任何没有开销的平均精度计算，我建议使用 keras callbacks.
您希望 oto 使用两个 callbacls 的主要想法 - EarlyStopping 和 ReduceLROnPlateau - 都作用于纪元结束并监视 loss 或 metric 值。他们从 method
logs 参数中得到这个值
def on_epoch_end(self, epoch, logs=None):
"""Called at the end of an epoch.
...
"""
- 将实际的 map 发送到日志值，我们强制此方法和所有从日志中获取精度值的回调使用它。 Callbcaks 从这里选择值（this ine int the code - early stopping, and this 一个用于 Reduce LR）。
因此，我们应该 "fake" 记录这两个回调。我想这不是理想的，但可行的解决方案。
这个 类 从回调继承并计算 map 值，他们还避免重新计算 map 通过共享对象 Hub。
from sklearn.metrics import average_precision_score
import keras
from keras.callbacks import Callback, EarlyStopping, ReduceLROnPlateau
class MAPHub:
def __init__(self):
self.map_value = None
- 它只是共享 map 值的中心。可能会引起一些副作用。你可以尽量避免使用它。
def on_epoch_end(self, epoch, logs):
"""self just a callbcak instance"""
if self.last_metric_for_epoch == epoch:
map_ = self.hub.map_value
else:
prediction = self.model.predict(self._data, verbose=1)
map_ = average_precision_score(self._target, prediction)
self.hub.map_value = map_
self.last_metric_for_epoch = epoch
- 此函数计算并共享 map
class EarlyStoppingByMAP(EarlyStopping):
def __init__(self, data, target, hub, *args, **kwargs):
"""
data, target - values and target for the map calculation
hub - shared object to store _map_ value
*args, **kwargs for the super __init__
"""
# I've set monitor to 'acc' here, because you're interested in metric, not loss
super(EarlyStoppingByMAP, self).__init__(monitor='acc', *args, **kwargs)
self._target = target
self._data = data
self.last_metric_for_epoch = -1
self.hub = hub
def on_epoch_end(self, epoch, logs):
"""
epoch is the number of epoch, logs is a dict logs with 'loss' value
and metric 'acc' values
"""
on_epoch_end(self, epoch, logs)
logs['acc'] = self.hub.map_value # "fake" metric with calculated value
print('Go callback from the {}, logs: \n{}'.format(EarlyStoppingByMAP.__name__, logs))
super(EarlyStoppingByMAP, self).on_epoch_end(epoch, logs) # works as a callback fn
class ReduceLROnPlateauByMAP(ReduceLROnPlateau):
def __init__(self, data, target, hub, *args, **kwargs):
# the same as in previous
# I've set monitor to 'acc' here, because you're interested in metric, not loss
super(ReduceLROnPlateauByMAP, self).__init__(monitor='acc', *args, **kwargs)
self._target = target
self._data = data
self.last_metric_for_epoch = -1
self.hub = hub
def on_epoch_end(self, epoch, logs):
on_epoch_end(self, epoch, logs)
logs['acc'] = self.hub.map_value # "fake" metric with calculated value
print('Go callback from the {}, logs: \n{}'.format(ReduceLROnPlateau.__name__, logs))
super(ReduceLROnPlateauByMAP, self).on_epoch_end(epoch, logs) # works as a callback fn
- 注意 不要在构造函数中使用 monitor 参数！您应该使用 'acc'，参数已设置为正确的值。
一些测试：
from keras.datasets import mnist
from keras.models import Model
from keras.layers import Dense, Input
import numpy as np
(X_tr, y_tr), (X_te, y_te) = mnist.load_data()
X_tr = (X_tr / 255.).reshape((60000, 784))
X_te = (X_te / 255.).reshape((10000, 784))
def binarize_labels(y):
y_bin = np.zeros((len(y), len(np.unique(y))))
y_bin[range(len(y)), y] = 1
return y_bin
y_train_bin, y_test_bin = binarize_labels(y_tr), binarize_labels(y_te)
inp = Input(shape=(784,))
x = Dense(784, activation='relu')(inp)
x = Dense(256, activation='relu')(x)
out = Dense(10, activation='softmax')(x)
model = Model(inp, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')
- 一个简单的 "test suite"。现在去适应它：
hub = MAPHub() # instentiate a hub
# I will use default params except patience as example, set it to 1 and 5
early_stop = EarlyStoppingByMAP(X_te, y_test_bin, hub, patience=1) # Patience is EarlyStopping's param
reduce_lt = ReduceLROnPlateauByMAP(X_te, y_test_bin, hub, patience=5) # Patience is ReduceLR's param
history = model.fit(X_tr, y_train_bin, epochs=10, callbacks=[early_stop, reduce_lt])
Out:
Epoch 1/10
60000/60000 [==============================] - 12s 207us/step - loss: 0.1815
10000/10000 [==============================] - 1s 59us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.18147853660446903, 'acc': 0.9934216252519924}
10000/10000 [==============================] - 0s 40us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.18147853660446903, 'acc': 0.9934216252519924}
Epoch 2/10
60000/60000 [==============================] - 12s 197us/step - loss: 0.0784
10000/10000 [==============================] - 0s 40us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.07844233275586739, 'acc': 0.9962269038764738}
10000/10000 [==============================] - 0s 41us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.07844233275586739, 'acc': 0.9962269038764738}
Epoch 3/10
60000/60000 [==============================] - 12s 197us/step - loss: 0.0556
10000/10000 [==============================] - 0s 40us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.05562876497630107, 'acc': 0.9972085346550085}
10000/10000 [==============================] - 0s 40us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.05562876497630107, 'acc': 0.9972085346550085}
Epoch 4/10
60000/60000 [==============================] - 12s 198us/step - loss: 0.0389
10000/10000 [==============================] - 0s 41us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.0388911374788188, 'acc': 0.9972696414934574}
10000/10000 [==============================] - 0s 41us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.0388911374788188, 'acc': 0.9972696414934574}
Epoch 5/10
60000/60000 [==============================] - 12s 197us/step - loss: 0.0330
10000/10000 [==============================] - 0s 39us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.03298293751536124, 'acc': 0.9959456176387349}
10000/10000 [==============================] - 0s 39us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.03298293751536124, 'acc': 0.9959456176387349}
好的，看起来至少可以提前停止。我想，ReduceLROnPlateau 因为他们使用相同的日志和相似的逻辑 - 如果设置了适当的参数。
如果你不想使用sklearn函数，但是SSDEvaluation（我就是找不到它是什么）——你可以很容易地采用on_epoch_method函数来处理这个评估功能。
希望对您有所帮助。
您可以使用 LambdaCallback 来更新您的 logs 对象：
假设你的 evaluation.evaluate() returns 像 {'val/mAP': value} 这样的字典，你可以这样做：
eval_callback = LambdaCallback(
on_epoch_end=lambda epoch, logs: logs.update(evaluation.evaluate())
)
这里的技巧是 logs 将进一步传递给其他回调，因此它们可以直接访问值：
early_stopping = EarlyStopping(monitor='val/mAP', min_delta=0.0, patience=10, verbose=1, mode='max')
它将自动出现在 CSVLogger 和任何其他回调中。但请注意 eval_callback 必须在使用回调列表中的值的任何回调之前：
callbacks = [eval_callback, early_stopping]