如何屏蔽具有 RepeatVector() 层的 LSTM 自动编码器中的输入？
How to mask the inputs in an LSTM autoencoder having a RepeatVector() layer?
我一直在尝试使用 LSTM 自动编码器获取向量序列的向量表示，以便我可以使用 SVM 或其他此类监督算法对序列进行分类。数据量使我无法使用完全连接的密集层进行分类。
我的输入的最短大小是 7 个时间步长，最长的序列是 356 个时间步长。因此，我用零填充较短的序列以获得最终 x_train 形状 (1326, 356, 8)，其中 1326 是训练样本的数量，8 是一个时间步长的维度。我正在尝试使用给定的 LSTM 自动编码器将这些序列编码为单个向量。
model.add(Masking(mask_value=0.0, input_shape=(max_len, 8)))
model.add(LSTM(100, activation='relu'))
model.add(RepeatVector(max_len))
model.add(LSTM(8, activation='relu', return_sequences=True))
model.compile(optimizer='adam', loss='mse')
model.fit(x_train, x_train, batch_size=32, callbacks=[chk], epochs=1000, validation_split=0.05, shuffle=True)
我正在尝试屏蔽零填充结果，但 RepeatVector() 层可能会阻碍该过程。因此，一段时间后均方误差损失变为 nan。谁能帮我解决如何在计算损失函数时只包含相关时间步而忽略其他时间步？
Keras 中的每一层都有一个 input_mask 和 output_mask，在您的示例中，掩码在第一个 LSTM 层之后（当 return_sequence = False 时）已经丢失。让我在下面的例子中解释这一点，并展示在 LSTM-autoencoder 中实现掩码的 2 种解决方案。
time_steps = 3
n_features = 2
input_layer = tfkl.Input(shape=(time_steps, n_features))
# I want to mask the timestep where all the feature values are 1 (usually we pad by 0)
x = tfk.layers.Masking(mask_value=1)(input_layer)
x = tfkl.LSTM(2, return_sequences=True)(x)
x = tfkl.LSTM(2, return_sequences=False)(x)
x = tfkl.RepeatVector(time_steps)(x)
x = tfkl.LSTM(2, return_sequences=True)(x)
x = tfkl.LSTM(2, return_sequences=True)(x)
x = tfk.layers.Dense(n_features)(x)
lstm_ae = tfk.models.Model(inputs=input_layer, outputs=x)
lstm_ae.compile(optimizer='adam', loss='mse')
print(lstm_ae.summary())
Model: "model_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_3 (InputLayer) [(None, 3, 2)] 0
_________________________________________________________________
masking_2 (Masking) (None, 3, 2) 0
_________________________________________________________________
lstm_8 (LSTM) (None, 3, 2) 40
_________________________________________________________________
lstm_9 (LSTM) (None, 2) 40
_________________________________________________________________
repeat_vector_2 (RepeatVecto (None, 3, 2) 0
_________________________________________________________________
lstm_10 (LSTM) (None, 3, 2) 40
_________________________________________________________________
lstm_11 (LSTM) (None, 3, 2) 40
_________________________________________________________________
dense_2 (Dense) (None, 3, 2) 6
=================================================================
Total params: 166
Trainable params: 166
Non-trainable params: 0
_________________________________________________________________
for i, l in enumerate(lstm_ae.layers):
print(f'layer {i}: {l}')
print(f'has input mask: {l.input_mask}')
print(f'has output mask: {l.output_mask}')
layer 0: <tensorflow.python.keras.engine.input_layer.InputLayer object at 0x645b49cf8>
has input mask: None
has output mask: None
layer 1: <tensorflow.python.keras.layers.core.Masking object at 0x645b49c88>
has input mask: None
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 2: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x645b4d0b8>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 3: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x645b4dba8>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: None
layer 4: <tensorflow.python.keras.layers.core.RepeatVector object at 0x645db0390>
has input mask: None
has output mask: None
layer 5: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x6470b5da0>
has input mask: None
has output mask: None
layer 6: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x6471410f0>
has input mask: None
has output mask: None
layer 7: <tensorflow.python.keras.layers.core.Dense object at 0x647dfdf60>
has input mask: None
has output mask: None
正如你在上面看到的，第二个LSTM层（return_sequence=False）returns一个None，这是有道理的，因为时间步丢失了（形状改变了) 并且图层不知道如何通过掩码，你也可以查看源代码你会看到它 returns 如果 return_sequence=True input_mask，否则 None.另一个问题当然是 RepeatVector 层，该层根本不支持显式屏蔽，这也是因为形状发生了变化。除了这个瓶颈部分（第二个LSTM + RepeatVector），模型的其他部分都可以通过mask，所以我们只需要处理瓶颈部分。
这里有2种可能的解决方案，我也会根据计算损失来验证。
第一个解决方案：通过传递 sample_weight
显式忽略时间步长
# last timestep should be masked because all feature values are 1
x = np.array([1, 2, 1, 2, 1, 1], dtype='float32').reshape(1, 3, 2)
print(x)
array([[[1., 2.],
[1., 2.],
[1., 1.]]], dtype=float32)
y = lstm_ae.predict(x)
print(y)
array([[[0.00020542, 0.00011909],
[0.0007361 , 0.00047323],
[0.00158514, 0.00107504]]], dtype=float32)
# the expected loss should be the sum of square error between the first 2 timesteps
# (2 features each timestep) divided by 6. you might expect that this should be
# divided by 4, but in the source code this is actually divided by 6, which doesn't
# matter a lot because only the gradient of loss matter, but not the loss itself.
expected_loss = np.square(x[:, :2, :] - y[:, :2, :]).sum()/6
print(expected_loss)
1.665958086649577
actual_loss_with_masking = lstm_ae.evaluate(x=x, y=x)
print(actual_loss_with_masking)
1.9984053373336792
# the actual loss still includes the last timestep, which means the masking is not # effectively passed to the output layer for calculating the loss
print(np.square(x-y).sum()/6)
1.9984052975972493
# if we provide the sample_weight 0 for each timestep that we want to mask, the
# loss will be ignored correctly
lstm_ae.compile(optimizer='adam', loss='mse', sample_weight_mode='temporal')
sample_weight_array = np.array([1, 1, 0]).reshape(1, 3) # it means to ignore the last timestep
actual_loss_with_sample_weight = lstm_ae.evaluate(x=x, y=x, sample_weight=sample_weight_array)
# the actual loss now is correct
print(actual_loss_with_sample_weight)
1.665958046913147
方案二：自定义瓶颈层手动过mask
class lstm_bottleneck(tf.keras.layers.Layer):
def __init__(self, lstm_units, time_steps, **kwargs):
self.lstm_units = lstm_units
self.time_steps = time_steps
self.lstm_layer = tfkl.LSTM(lstm_units, return_sequences=False)
self.repeat_layer = tfkl.RepeatVector(time_steps)
super(lstm_bottleneck, self).__init__(**kwargs)
def call(self, inputs):
# just call the two initialized layers
return self.repeat_layer(self.lstm_layer(inputs))
def compute_mask(self, inputs, mask=None):
# return the input_mask directly
return mask
time_steps = 3
n_features = 2
input_layer = tfkl.Input(shape=(time_steps, n_features))
# I want to mask the timestep where all the feature values are 1 (usually we pad by 0)
x = tfk.layers.Masking(mask_value=1)(input_layer)
x = tfkl.LSTM(2, return_sequences=True)(x)
x = lstm_bottleneck(lstm_units=2, time_steps=3)(x)
# x = tfkl.LSTM(2, return_sequences=False)(x)
# x = tfkl.RepeatVector(time_steps)(x)
x = tfkl.LSTM(2, return_sequences=True)(x)
x = tfkl.LSTM(2, return_sequences=True)(x)
x = tfk.layers.Dense(n_features)(x)
lstm_ae = tfk.models.Model(inputs=input_layer, outputs=x)
lstm_ae.compile(optimizer='adam', loss='mse')
print(lstm_ae.summary())
Model: "model_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_3 (InputLayer) [(None, 3, 2)] 0
_________________________________________________________________
masking_2 (Masking) (None, 3, 2) 0
_________________________________________________________________
lstm_10 (LSTM) (None, 3, 2) 40
_________________________________________________________________
lstm_bottleneck_3 (lstm_bott (None, 3, 2) 40
_________________________________________________________________
lstm_12 (LSTM) (None, 3, 2) 40
_________________________________________________________________
lstm_13 (LSTM) (None, 3, 2) 40
_________________________________________________________________
dense_2 (Dense) (None, 3, 2) 6
=================================================================
Total params: 166
Trainable params: 166
Non-trainable params: 0
_________________________________________________________________
for i, l in enumerate(lstm_ae.layers):
print(f'layer {i}: {l}')
print(f'has input mask: {l.input_mask}')
print(f'has output mask: {l.output_mask}')
layer 0: <tensorflow.python.keras.engine.input_layer.InputLayer object at 0x64dbf98d0>
has input mask: None
has output mask: None
layer 1: <tensorflow.python.keras.layers.core.Masking object at 0x64dbf9f60>
has input mask: None
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 2: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x64dbf9550>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 3: <__main__.lstm_bottleneck object at 0x64dbf91d0>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 4: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x64e04ca20>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 5: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x64eeb8b00>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 6: <tensorflow.python.keras.layers.core.Dense object at 0x64ef43208>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
正如我们已经看到的，掩码现在已成功传递到输出层。我们还将验证损失不包括掩码时间步长。
# last timestep should be masked because all feature values are 1
x = np.array([1, 2, 1, 2, 1, 1], dtype='float32').reshape(1, 3, 2)
print(x)
array([[[1., 2.],
[1., 2.],
[1., 1.]]], dtype=float32)
y = lstm_ae.predict(x)
print(y)
array([[[ 0.00065455, -0.00294413],
[ 0.00166675, -0.00742249],
[ 0.00166675, -0.00742249]]], dtype=float32)
# the expected loss should be the square error between the first 2 timesteps divided by 6
expected_loss = np.square(x[:, :2, :] - y[:, :2, :]).sum()/6
print(expected_loss)
1.672815163930257
# now the loss is correct with a custom layer
actual_loss_with_masking = lstm_ae.evaluate(x=x, y=x)
print(actual_loss_with_masking)
1.672815203666687
我一直在尝试使用 LSTM 自动编码器获取向量序列的向量表示，以便我可以使用 SVM 或其他此类监督算法对序列进行分类。数据量使我无法使用完全连接的密集层进行分类。
我的输入的最短大小是 7 个时间步长，最长的序列是 356 个时间步长。因此，我用零填充较短的序列以获得最终 x_train 形状 (1326, 356, 8)，其中 1326 是训练样本的数量，8 是一个时间步长的维度。我正在尝试使用给定的 LSTM 自动编码器将这些序列编码为单个向量。
model.add(Masking(mask_value=0.0, input_shape=(max_len, 8)))
model.add(LSTM(100, activation='relu'))
model.add(RepeatVector(max_len))
model.add(LSTM(8, activation='relu', return_sequences=True))
model.compile(optimizer='adam', loss='mse')
model.fit(x_train, x_train, batch_size=32, callbacks=[chk], epochs=1000, validation_split=0.05, shuffle=True)
我正在尝试屏蔽零填充结果，但 RepeatVector() 层可能会阻碍该过程。因此，一段时间后均方误差损失变为 nan。谁能帮我解决如何在计算损失函数时只包含相关时间步而忽略其他时间步？
Keras 中的每一层都有一个 input_mask 和 output_mask，在您的示例中，掩码在第一个 LSTM 层之后（当 return_sequence = False 时）已经丢失。让我在下面的例子中解释这一点，并展示在 LSTM-autoencoder 中实现掩码的 2 种解决方案。
time_steps = 3
n_features = 2
input_layer = tfkl.Input(shape=(time_steps, n_features))
# I want to mask the timestep where all the feature values are 1 (usually we pad by 0)
x = tfk.layers.Masking(mask_value=1)(input_layer)
x = tfkl.LSTM(2, return_sequences=True)(x)
x = tfkl.LSTM(2, return_sequences=False)(x)
x = tfkl.RepeatVector(time_steps)(x)
x = tfkl.LSTM(2, return_sequences=True)(x)
x = tfkl.LSTM(2, return_sequences=True)(x)
x = tfk.layers.Dense(n_features)(x)
lstm_ae = tfk.models.Model(inputs=input_layer, outputs=x)
lstm_ae.compile(optimizer='adam', loss='mse')
print(lstm_ae.summary())
Model: "model_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_3 (InputLayer) [(None, 3, 2)] 0
_________________________________________________________________
masking_2 (Masking) (None, 3, 2) 0
_________________________________________________________________
lstm_8 (LSTM) (None, 3, 2) 40
_________________________________________________________________
lstm_9 (LSTM) (None, 2) 40
_________________________________________________________________
repeat_vector_2 (RepeatVecto (None, 3, 2) 0
_________________________________________________________________
lstm_10 (LSTM) (None, 3, 2) 40
_________________________________________________________________
lstm_11 (LSTM) (None, 3, 2) 40
_________________________________________________________________
dense_2 (Dense) (None, 3, 2) 6
=================================================================
Total params: 166
Trainable params: 166
Non-trainable params: 0
_________________________________________________________________
for i, l in enumerate(lstm_ae.layers):
print(f'layer {i}: {l}')
print(f'has input mask: {l.input_mask}')
print(f'has output mask: {l.output_mask}')
layer 0: <tensorflow.python.keras.engine.input_layer.InputLayer object at 0x645b49cf8>
has input mask: None
has output mask: None
layer 1: <tensorflow.python.keras.layers.core.Masking object at 0x645b49c88>
has input mask: None
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 2: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x645b4d0b8>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 3: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x645b4dba8>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: None
layer 4: <tensorflow.python.keras.layers.core.RepeatVector object at 0x645db0390>
has input mask: None
has output mask: None
layer 5: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x6470b5da0>
has input mask: None
has output mask: None
layer 6: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x6471410f0>
has input mask: None
has output mask: None
layer 7: <tensorflow.python.keras.layers.core.Dense object at 0x647dfdf60>
has input mask: None
has output mask: None
正如你在上面看到的，第二个LSTM层（return_sequence=False）returns一个None，这是有道理的，因为时间步丢失了（形状改变了) 并且图层不知道如何通过掩码，你也可以查看源代码你会看到它 returns 如果 return_sequence=True input_mask，否则 None.另一个问题当然是 RepeatVector 层，该层根本不支持显式屏蔽，这也是因为形状发生了变化。除了这个瓶颈部分（第二个LSTM + RepeatVector），模型的其他部分都可以通过mask，所以我们只需要处理瓶颈部分。
这里有2种可能的解决方案，我也会根据计算损失来验证。
第一个解决方案：通过传递 sample_weight
显式忽略时间步长# last timestep should be masked because all feature values are 1
x = np.array([1, 2, 1, 2, 1, 1], dtype='float32').reshape(1, 3, 2)
print(x)
array([[[1., 2.],
[1., 2.],
[1., 1.]]], dtype=float32)
y = lstm_ae.predict(x)
print(y)
array([[[0.00020542, 0.00011909],
[0.0007361 , 0.00047323],
[0.00158514, 0.00107504]]], dtype=float32)
# the expected loss should be the sum of square error between the first 2 timesteps
# (2 features each timestep) divided by 6. you might expect that this should be
# divided by 4, but in the source code this is actually divided by 6, which doesn't
# matter a lot because only the gradient of loss matter, but not the loss itself.
expected_loss = np.square(x[:, :2, :] - y[:, :2, :]).sum()/6
print(expected_loss)
1.665958086649577
actual_loss_with_masking = lstm_ae.evaluate(x=x, y=x)
print(actual_loss_with_masking)
1.9984053373336792
# the actual loss still includes the last timestep, which means the masking is not # effectively passed to the output layer for calculating the loss
print(np.square(x-y).sum()/6)
1.9984052975972493
# if we provide the sample_weight 0 for each timestep that we want to mask, the
# loss will be ignored correctly
lstm_ae.compile(optimizer='adam', loss='mse', sample_weight_mode='temporal')
sample_weight_array = np.array([1, 1, 0]).reshape(1, 3) # it means to ignore the last timestep
actual_loss_with_sample_weight = lstm_ae.evaluate(x=x, y=x, sample_weight=sample_weight_array)
# the actual loss now is correct
print(actual_loss_with_sample_weight)
1.665958046913147
方案二：自定义瓶颈层手动过mask
class lstm_bottleneck(tf.keras.layers.Layer):
def __init__(self, lstm_units, time_steps, **kwargs):
self.lstm_units = lstm_units
self.time_steps = time_steps
self.lstm_layer = tfkl.LSTM(lstm_units, return_sequences=False)
self.repeat_layer = tfkl.RepeatVector(time_steps)
super(lstm_bottleneck, self).__init__(**kwargs)
def call(self, inputs):
# just call the two initialized layers
return self.repeat_layer(self.lstm_layer(inputs))
def compute_mask(self, inputs, mask=None):
# return the input_mask directly
return mask
time_steps = 3
n_features = 2
input_layer = tfkl.Input(shape=(time_steps, n_features))
# I want to mask the timestep where all the feature values are 1 (usually we pad by 0)
x = tfk.layers.Masking(mask_value=1)(input_layer)
x = tfkl.LSTM(2, return_sequences=True)(x)
x = lstm_bottleneck(lstm_units=2, time_steps=3)(x)
# x = tfkl.LSTM(2, return_sequences=False)(x)
# x = tfkl.RepeatVector(time_steps)(x)
x = tfkl.LSTM(2, return_sequences=True)(x)
x = tfkl.LSTM(2, return_sequences=True)(x)
x = tfk.layers.Dense(n_features)(x)
lstm_ae = tfk.models.Model(inputs=input_layer, outputs=x)
lstm_ae.compile(optimizer='adam', loss='mse')
print(lstm_ae.summary())
Model: "model_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_3 (InputLayer) [(None, 3, 2)] 0
_________________________________________________________________
masking_2 (Masking) (None, 3, 2) 0
_________________________________________________________________
lstm_10 (LSTM) (None, 3, 2) 40
_________________________________________________________________
lstm_bottleneck_3 (lstm_bott (None, 3, 2) 40
_________________________________________________________________
lstm_12 (LSTM) (None, 3, 2) 40
_________________________________________________________________
lstm_13 (LSTM) (None, 3, 2) 40
_________________________________________________________________
dense_2 (Dense) (None, 3, 2) 6
=================================================================
Total params: 166
Trainable params: 166
Non-trainable params: 0
_________________________________________________________________
for i, l in enumerate(lstm_ae.layers):
print(f'layer {i}: {l}')
print(f'has input mask: {l.input_mask}')
print(f'has output mask: {l.output_mask}')
layer 0: <tensorflow.python.keras.engine.input_layer.InputLayer object at 0x64dbf98d0>
has input mask: None
has output mask: None
layer 1: <tensorflow.python.keras.layers.core.Masking object at 0x64dbf9f60>
has input mask: None
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 2: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x64dbf9550>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 3: <__main__.lstm_bottleneck object at 0x64dbf91d0>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 4: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x64e04ca20>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 5: <tensorflow.python.keras.layers.recurrent_v2.LSTM object at 0x64eeb8b00>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
layer 6: <tensorflow.python.keras.layers.core.Dense object at 0x64ef43208>
has input mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
has output mask: Tensor("masking_2/Identity_1:0", shape=(None, 3), dtype=bool)
正如我们已经看到的，掩码现在已成功传递到输出层。我们还将验证损失不包括掩码时间步长。
# last timestep should be masked because all feature values are 1
x = np.array([1, 2, 1, 2, 1, 1], dtype='float32').reshape(1, 3, 2)
print(x)
array([[[1., 2.],
[1., 2.],
[1., 1.]]], dtype=float32)
y = lstm_ae.predict(x)
print(y)
array([[[ 0.00065455, -0.00294413],
[ 0.00166675, -0.00742249],
[ 0.00166675, -0.00742249]]], dtype=float32)
# the expected loss should be the square error between the first 2 timesteps divided by 6
expected_loss = np.square(x[:, :2, :] - y[:, :2, :]).sum()/6
print(expected_loss)
1.672815163930257
# now the loss is correct with a custom layer
actual_loss_with_masking = lstm_ae.evaluate(x=x, y=x)
print(actual_loss_with_masking)
1.672815203666687