为什么我在 TensorFlow Keras 中的损失函数和指标之间得到不同的值?
Why am I getting different values between loss functions and metrics in TensorFlow Keras?
在我使用 TensorFlow 进行的 CNN 训练中,我使用 Keras.losses.poisson
作为损失函数。现在,我喜欢计算损失函数旁边的许多指标,我观察到 Keras.metrics.poisson
给出了不同的结果——尽管两者是相同的函数。
查看此处的一些示例输出:loss
和 poisson
输出具有不同的范围,0.5 与 0.12:
Epoch 1/20
Epoch 00001: val_loss improved from inf to 0.53228, saving model to P:\Data\xyz.h5
- 8174s - loss: 0.5085 - binary_crossentropy: 0.1252 - poisson: 0.1271 - mean_squared_error: 1.2530e-04 - mean_absolute_error: 0.0035 - mean_absolute_percentage_error: 38671.1055 - val_loss: 0.5323 - val_binary_crossentropy: 0.1305 - val_poisson: 0.1331 - val_mean_squared_error: 5.8477e-05 - val_mean_absolute_error: 0.0035 - val_mean_absolute_percentage_error: 1617.8346
Epoch 2/20
Epoch 00002: val_loss improved from 0.53228 to 0.53218, saving model to P:\Data\xyz.h5
- 8042s - loss: 0.5067 - binary_crossentropy: 0.1246 - poisson: 0.1267 - mean_squared_error: 1.0892e-05 - mean_absolute_error: 0.0017 - mean_absolute_percentage_error: 410.8044 - val_loss: 0.5322 - val_binary_crossentropy: 0.1304 - val_poisson: 0.1330 - val_mean_squared_error: 4.9087e-05 - val_mean_absolute_error: 0.0035 - val_mean_absolute_percentage_error: 545.5222
Epoch 3/20
Epoch 00003: val_loss improved from 0.53218 to 0.53199, saving model to P:\Data\xyz.h5
- 8038s - loss: 0.5066 - binary_crossentropy: 0.1246 - poisson: 0.1266 - mean_squared_error: 6.6870e-06 - mean_absolute_error: 0.0013 - mean_absolute_percentage_error: 298.9844 - val_loss: 0.5320 - val_binary_crossentropy: 0.1304 - val_poisson: 0.1330 - val_mean_squared_error: 4.3858e-05 - val_mean_absolute_error: 0.0031 - val_mean_absolute_percentage_error: 452.3541
我在输入这个问题时发现了一个类似的问题:
但是,我没有使用正则化。
此外,我遇到过这个,至少帮助我重现了这个问题:Same function in Keras Loss and Metric give different values even without regularization
from tensorflow import keras
layer = keras.layers.Input(shape=(1, 1, 1))
model = keras.models.Model(inputs=layer, outputs=layer)
model.compile(optimizer='adam', loss='poisson', metrics=['poisson'])
data = [[[[[1]]], [[[2]]], [[[3]]]]]
model.fit(x=data, y=data, batch_size=2, verbose=1)
然后我发现基本上是维度触发了这个问题。从下面的扩展示例中,您可以看到
- 这个问题可以用很多损失函数重现(那些帽子不以
mean_
开头),
- 将
tensorflow.keras
替换为 keras
和 后问题消失
tensorflow.keras
如果数据的维数大于三,则似乎按批量大小缩放指标。至少这是我的拙见。
代码:
import numpy as np
from tensorflow import keras
# import keras
nSamples = 98765
nBatch = 2345
metric = 'poisson'
# metric = 'squared_hinge'
# metric = 'logcosh'
# metric = 'cosine_proximity'
# metric = 'binary_crossentropy'
# example data: always the same samples
np.random.seed(0)
dataIn = np.random.rand(nSamples)
dataOut = np.random.rand(nSamples)
for dataDim in range(1, 10):
# reshape samples into size (1,), ..., (1, 1, ...) according to dataDim
dataIn = np.expand_dims(dataIn, axis=-1)
dataOut = np.expand_dims(dataOut, axis=-1)
# build a model that does absolutely nothing
Layer = keras.layers.Input(shape=np.ones(dataDim))
model = keras.models.Model(inputs=Layer, outputs=Layer)
# compile, fit and observe loss ratio
model.compile(optimizer='adam', loss=metric, metrics=[metric])
history = model.fit(x=dataIn, y=dataOut, batch_size=nBatch, verbose=1)
lossRatio = history.history['loss'][0] / history.history[metric][0]
print(lossRatio)
我发现这种行为至少不一致。我应该将其视为错误还是功能?
更新:经过进一步调查,我发现指标值计算正确,而损失值不正确;事实上,损失是样本损失的加权和,其中每个样本的权重是样本所在批次的大小。这有两个含义:
- 如果批量大小除以样本数,则所有样本的重量都相同,损失简单地减去等于批量大小的那个因子。
- 如果批次大小不除以样本数,因为批次通常被打乱,权重和计算的损失从一个时期到下一个时期会发生变化,尽管其他任何事情都没有改变。这也适用于 MSE 等指标。
下面的代码证明了这几点:
import numpy as np
import tensorflow as tf
from tensorflow import keras
# metric = keras.metrics.poisson
# metricName = 'poisson'
metric = keras.metrics.mse
metricName = 'mean_squared_error'
nSamples = 3
nBatchSize = 2
dataIn = np.random.rand(nSamples, 1, 1, 1)
dataOut = np.random.rand(nSamples, 1, 1, 1)
tf.InteractiveSession()
layer = keras.layers.Input(shape=(1, 1, 1))
model = keras.models.Model(inputs=layer, outputs=layer)
model.compile(optimizer='adam', loss=metric, metrics=[metric])
h = model.fit(x=dataIn, y=dataOut, batch_size=nBatchSize, verbose=1, epochs=10)
for (historyMetric, historyLoss) in zip(h.history[metricName], h.history['loss']):
# the metric value is correct and can be reproduced in a number of ways
kerasMetricOfData = metric(dataOut, dataIn).eval()
averageMetric = np.mean(kerasMetricOfData)
assert np.isclose(historyMetric, averageMetric), "..."
flattenedMetric = metric(dataOut.flatten(), dataIn.flatten()).eval()
assert np.isclose(historyMetric, flattenedMetric), "..."
if metric == keras.metrics.poisson:
numpyMetric = np.mean(dataIn - np.log(dataIn) * dataOut)
assert np.isclose(historyMetric, numpyMetric), "..."
# the loss value is incorrect by at least a scaling factor (~ batch size).
# also varies *randomly* if the batch size does not divide the # of samples:
if nSamples == 3:
incorrectLoss = np.array([
np.mean(kerasMetricOfData.flatten() * [1, nBatchSize, nBatchSize]),
np.mean(kerasMetricOfData.flatten() * [nBatchSize, 1, nBatchSize]),
np.mean(kerasMetricOfData.flatten() * [nBatchSize, nBatchSize, 1]),
])
elif nSamples == 4:
incorrectLoss = np.mean(kerasMetricOfData) * nBatchSize
assert np.any(np.isclose(historyLoss, incorrectLoss)), "..."
它输出:
Epoch 1/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0044 - mean_squared_error: 0.0022
3/3 [==============================] - 0s 5ms/sample - loss: 0.0099 - mean_squared_error: 0.0084
Epoch 2/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0238 - mean_squared_error: 0.0119
3/3 [==============================] - 0s 2ms/sample - loss: 0.0163 - mean_squared_error: 0.0084
Epoch 3/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0238 - mean_squared_error: 0.0119
3/3 [==============================] - 0s 2ms/sample - loss: 0.0163 - mean_squared_error: 0.0084
Epoch 4/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0238 - mean_squared_error: 0.0119
3/3 [==============================] - 0s 2ms/sample - loss: 0.0163 - mean_squared_error: 0.0084
Epoch 5/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0238 - mean_squared_error: 0.0119
3/3 [==============================] - 0s 2ms/sample - loss: 0.0163 - mean_squared_error: 0.0084
Epoch 6/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0222 - mean_squared_error: 0.0111
3/3 [==============================] - 0s 2ms/sample - loss: 0.0158 - mean_squared_error: 0.0084
Epoch 7/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0222 - mean_squared_error: 0.0111
3/3 [==============================] - 0s 2ms/sample - loss: 0.0158 - mean_squared_error: 0.0084
Epoch 8/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0238 - mean_squared_error: 0.0119
3/3 [==============================] - 0s 2ms/sample - loss: 0.0163 - mean_squared_error: 0.0084
Epoch 9/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0222 - mean_squared_error: 0.0111
3/3 [==============================] - 0s 2ms/sample - loss: 0.0158 - mean_squared_error: 0.0084
Epoch 10/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0044 - mean_squared_error: 0.0022
3/3 [==============================] - 0s 2ms/sample - loss: 0.0099 - mean_squared_error: 0.0084
Update:最后,使用keras.metrics.mse
和'mse'
似乎有区别,如本例所示:
import numpy as np
from tensorflow import keras
# these three reproduce the issue:
# metric = keras.metrics.poisson
# metric = 'poisson'
# metric = keras.metrics.mse
# this one does not:
metric = 'mse'
nSamples = 3
nBatchSize = 2
dataIn = np.random.rand(nSamples, 1, 1, 1)
dataOut = np.random.rand(nSamples, 1, 1, 1)
layer = keras.layers.Input(shape=(1, 1, 1))
model = keras.models.Model(inputs=layer, outputs=layer)
model.compile(optimizer='adam', loss=metric, metrics=[metric])
model.fit(x=dataIn, y=dataOut, batch_size=2, verbose=1, epochs=10)
我开始相信这一定是一个错误,reported it here。
这已被确认为错误并已修复。
有关详细信息,请参阅 https://github.com/tensorflow/tensorflow/issues/25970。
在我使用 TensorFlow 进行的 CNN 训练中,我使用 Keras.losses.poisson
作为损失函数。现在,我喜欢计算损失函数旁边的许多指标,我观察到 Keras.metrics.poisson
给出了不同的结果——尽管两者是相同的函数。
查看此处的一些示例输出:loss
和 poisson
输出具有不同的范围,0.5 与 0.12:
Epoch 1/20
Epoch 00001: val_loss improved from inf to 0.53228, saving model to P:\Data\xyz.h5
- 8174s - loss: 0.5085 - binary_crossentropy: 0.1252 - poisson: 0.1271 - mean_squared_error: 1.2530e-04 - mean_absolute_error: 0.0035 - mean_absolute_percentage_error: 38671.1055 - val_loss: 0.5323 - val_binary_crossentropy: 0.1305 - val_poisson: 0.1331 - val_mean_squared_error: 5.8477e-05 - val_mean_absolute_error: 0.0035 - val_mean_absolute_percentage_error: 1617.8346
Epoch 2/20
Epoch 00002: val_loss improved from 0.53228 to 0.53218, saving model to P:\Data\xyz.h5
- 8042s - loss: 0.5067 - binary_crossentropy: 0.1246 - poisson: 0.1267 - mean_squared_error: 1.0892e-05 - mean_absolute_error: 0.0017 - mean_absolute_percentage_error: 410.8044 - val_loss: 0.5322 - val_binary_crossentropy: 0.1304 - val_poisson: 0.1330 - val_mean_squared_error: 4.9087e-05 - val_mean_absolute_error: 0.0035 - val_mean_absolute_percentage_error: 545.5222
Epoch 3/20
Epoch 00003: val_loss improved from 0.53218 to 0.53199, saving model to P:\Data\xyz.h5
- 8038s - loss: 0.5066 - binary_crossentropy: 0.1246 - poisson: 0.1266 - mean_squared_error: 6.6870e-06 - mean_absolute_error: 0.0013 - mean_absolute_percentage_error: 298.9844 - val_loss: 0.5320 - val_binary_crossentropy: 0.1304 - val_poisson: 0.1330 - val_mean_squared_error: 4.3858e-05 - val_mean_absolute_error: 0.0031 - val_mean_absolute_percentage_error: 452.3541
我在输入这个问题时发现了一个类似的问题:
此外,我遇到过这个,至少帮助我重现了这个问题:Same function in Keras Loss and Metric give different values even without regularization
from tensorflow import keras
layer = keras.layers.Input(shape=(1, 1, 1))
model = keras.models.Model(inputs=layer, outputs=layer)
model.compile(optimizer='adam', loss='poisson', metrics=['poisson'])
data = [[[[[1]]], [[[2]]], [[[3]]]]]
model.fit(x=data, y=data, batch_size=2, verbose=1)
然后我发现基本上是维度触发了这个问题。从下面的扩展示例中,您可以看到
- 这个问题可以用很多损失函数重现(那些帽子不以
mean_
开头), - 将
tensorflow.keras
替换为keras
和 后问题消失
tensorflow.keras
如果数据的维数大于三,则似乎按批量大小缩放指标。至少这是我的拙见。
代码:
import numpy as np
from tensorflow import keras
# import keras
nSamples = 98765
nBatch = 2345
metric = 'poisson'
# metric = 'squared_hinge'
# metric = 'logcosh'
# metric = 'cosine_proximity'
# metric = 'binary_crossentropy'
# example data: always the same samples
np.random.seed(0)
dataIn = np.random.rand(nSamples)
dataOut = np.random.rand(nSamples)
for dataDim in range(1, 10):
# reshape samples into size (1,), ..., (1, 1, ...) according to dataDim
dataIn = np.expand_dims(dataIn, axis=-1)
dataOut = np.expand_dims(dataOut, axis=-1)
# build a model that does absolutely nothing
Layer = keras.layers.Input(shape=np.ones(dataDim))
model = keras.models.Model(inputs=Layer, outputs=Layer)
# compile, fit and observe loss ratio
model.compile(optimizer='adam', loss=metric, metrics=[metric])
history = model.fit(x=dataIn, y=dataOut, batch_size=nBatch, verbose=1)
lossRatio = history.history['loss'][0] / history.history[metric][0]
print(lossRatio)
我发现这种行为至少不一致。我应该将其视为错误还是功能?
更新:经过进一步调查,我发现指标值计算正确,而损失值不正确;事实上,损失是样本损失的加权和,其中每个样本的权重是样本所在批次的大小。这有两个含义:
- 如果批量大小除以样本数,则所有样本的重量都相同,损失简单地减去等于批量大小的那个因子。
- 如果批次大小不除以样本数,因为批次通常被打乱,权重和计算的损失从一个时期到下一个时期会发生变化,尽管其他任何事情都没有改变。这也适用于 MSE 等指标。
下面的代码证明了这几点:
import numpy as np
import tensorflow as tf
from tensorflow import keras
# metric = keras.metrics.poisson
# metricName = 'poisson'
metric = keras.metrics.mse
metricName = 'mean_squared_error'
nSamples = 3
nBatchSize = 2
dataIn = np.random.rand(nSamples, 1, 1, 1)
dataOut = np.random.rand(nSamples, 1, 1, 1)
tf.InteractiveSession()
layer = keras.layers.Input(shape=(1, 1, 1))
model = keras.models.Model(inputs=layer, outputs=layer)
model.compile(optimizer='adam', loss=metric, metrics=[metric])
h = model.fit(x=dataIn, y=dataOut, batch_size=nBatchSize, verbose=1, epochs=10)
for (historyMetric, historyLoss) in zip(h.history[metricName], h.history['loss']):
# the metric value is correct and can be reproduced in a number of ways
kerasMetricOfData = metric(dataOut, dataIn).eval()
averageMetric = np.mean(kerasMetricOfData)
assert np.isclose(historyMetric, averageMetric), "..."
flattenedMetric = metric(dataOut.flatten(), dataIn.flatten()).eval()
assert np.isclose(historyMetric, flattenedMetric), "..."
if metric == keras.metrics.poisson:
numpyMetric = np.mean(dataIn - np.log(dataIn) * dataOut)
assert np.isclose(historyMetric, numpyMetric), "..."
# the loss value is incorrect by at least a scaling factor (~ batch size).
# also varies *randomly* if the batch size does not divide the # of samples:
if nSamples == 3:
incorrectLoss = np.array([
np.mean(kerasMetricOfData.flatten() * [1, nBatchSize, nBatchSize]),
np.mean(kerasMetricOfData.flatten() * [nBatchSize, 1, nBatchSize]),
np.mean(kerasMetricOfData.flatten() * [nBatchSize, nBatchSize, 1]),
])
elif nSamples == 4:
incorrectLoss = np.mean(kerasMetricOfData) * nBatchSize
assert np.any(np.isclose(historyLoss, incorrectLoss)), "..."
它输出:
Epoch 1/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0044 - mean_squared_error: 0.0022
3/3 [==============================] - 0s 5ms/sample - loss: 0.0099 - mean_squared_error: 0.0084
Epoch 2/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0238 - mean_squared_error: 0.0119
3/3 [==============================] - 0s 2ms/sample - loss: 0.0163 - mean_squared_error: 0.0084
Epoch 3/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0238 - mean_squared_error: 0.0119
3/3 [==============================] - 0s 2ms/sample - loss: 0.0163 - mean_squared_error: 0.0084
Epoch 4/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0238 - mean_squared_error: 0.0119
3/3 [==============================] - 0s 2ms/sample - loss: 0.0163 - mean_squared_error: 0.0084
Epoch 5/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0238 - mean_squared_error: 0.0119
3/3 [==============================] - 0s 2ms/sample - loss: 0.0163 - mean_squared_error: 0.0084
Epoch 6/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0222 - mean_squared_error: 0.0111
3/3 [==============================] - 0s 2ms/sample - loss: 0.0158 - mean_squared_error: 0.0084
Epoch 7/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0222 - mean_squared_error: 0.0111
3/3 [==============================] - 0s 2ms/sample - loss: 0.0158 - mean_squared_error: 0.0084
Epoch 8/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0238 - mean_squared_error: 0.0119
3/3 [==============================] - 0s 2ms/sample - loss: 0.0163 - mean_squared_error: 0.0084
Epoch 9/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0222 - mean_squared_error: 0.0111
3/3 [==============================] - 0s 2ms/sample - loss: 0.0158 - mean_squared_error: 0.0084
Epoch 10/10
2/3 [===================>..........] - ETA: 0s - loss: 0.0044 - mean_squared_error: 0.0022
3/3 [==============================] - 0s 2ms/sample - loss: 0.0099 - mean_squared_error: 0.0084
Update:最后,使用keras.metrics.mse
和'mse'
似乎有区别,如本例所示:
import numpy as np
from tensorflow import keras
# these three reproduce the issue:
# metric = keras.metrics.poisson
# metric = 'poisson'
# metric = keras.metrics.mse
# this one does not:
metric = 'mse'
nSamples = 3
nBatchSize = 2
dataIn = np.random.rand(nSamples, 1, 1, 1)
dataOut = np.random.rand(nSamples, 1, 1, 1)
layer = keras.layers.Input(shape=(1, 1, 1))
model = keras.models.Model(inputs=layer, outputs=layer)
model.compile(optimizer='adam', loss=metric, metrics=[metric])
model.fit(x=dataIn, y=dataOut, batch_size=2, verbose=1, epochs=10)
我开始相信这一定是一个错误,reported it here。
这已被确认为错误并已修复。 有关详细信息,请参阅 https://github.com/tensorflow/tensorflow/issues/25970。