保存n个之前的训练权重
Save n previous weights of training
我想在 Tensorflow 中编写一个自定义层，我需要在其中保存该层的先前权重（因此对于最后 n 个时期中的每一个）。我会在自定义图层的哪个位置执行此操作？
例如，这是自定义层在 tensorflow 示例中的样子：
class MyDenseLayer(tf.keras.layers.Layer):
def __init__(self, num_outputs):
super(MyDenseLayer, self).__init__()
self.num_outputs = num_outputs
def build(self, input_shape):
self.kernel = self.add_weight("kernel",
shape=[int(input_shape[-1]),
self.num_outputs])
def call(self, input):
return tf.matmul(input, self.kernel)
call() 每个批次调用方法。
在 __init__() 内添加计数器并在 call() 内保存：
class MyDenseLayer(tf.keras.layers.Layer):
def __init__(self, num_outputs):
super(MyDenseLayer, self).__init__()
self.num_outputs = num_outputs
self.counter = 0
def call(self, input):
self.counter += 1
if self.counter % batches_per_epoch == 0:
# add saving here
return tf.matmul(input, self.kernel)
这是我的方法：
1- 使用 ModelCheckpoint 回调
这里我创建了一个前馈神经网络。然后，我使用 tf.keras.callbacks.ModelCheckpoint 在每个时期保存模型。最后，我加载一个保存的模型并访问它的权重。
-首先，让我们创建一个简单的前馈神经网络（当然，您可以使用任何其他层或模型）：
import tensorflow as tf
from tensorflow.keras.layers import Dense
from tensorflow.keras import Sequential
from tensorflow.keras.callbacks import ModelCheckpoint
from sklearn.datasets import make_blobs
from tensorflow.keras.utils import to_categorical
model = Sequential()
model.add(Dense(units=3 , input_dim=5 , activation='relu', name='Dense_1'))
model.add(Dense(units=2 , activation='softmax', name='Dense_2'))
model.summary()
-这里是摘要输出：
Model: "sequential_6"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
Dense_1 (Dense) (None, 3) 18
_________________________________________________________________
Dense_2 (Dense) (None, 2) 8
=================================================================
Total params: 26
Trainable params: 26
Non-trainable params: 0
_________________________________________________________________
-创建用于训练模型的虚拟数据：
train_x , train_y = make_blobs(n_samples=1000, centers=2, n_features=5)
train_y = to_categorical(train_y,2)
-编译并训练模型：
LOG_DIRECTORY = './Whosebug/'
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model_checkpoint = ModelCheckpoint(LOG_DIRECTORY+'weights{epoch:03d}.h5',
save_freq='epoch',
verbose=1)
model.fit(train_x, train_y,
batch_size=32,
epochs=100,
verbose=1,
callbacks=[model_checkpoint]
)
-让我们看看一些 fit 输出：
Epoch 1/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
Epoch 00001: saving model to ./Whosebug/weights001.h5
Epoch 2/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
Epoch 00002: saving model to ./Whosebug/weights002.h5
Epoch 3/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
Epoch 00003: saving model to ./Whosebug/weights003.h5
Epoch 4/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
Epoch 00004: saving model to ./Whosebug/weights004.h5
正如您在上面的输出中看到的，模型在每个时期后使用 model_checkpoint = ModelCheckpoint ....
保存
-加载保存的模型：
from tensorflow.keras.models import load_model
saved_model =load_model(LOG_DIRECTORY+'weights097.h5')
-打印层的重量：
print(saved_model.layers[0].weights)
在您的情况下，使用所需图层的索引。
-输出：
输出是 tf.Variable 个权重和偏差
的列表
[<tf.Variable 'Dense_1/kernel:0' shape=(5, 3) dtype=float32, numpy=
array([[ 0.44274166, -0.46638554, -0.40543374],
[-0.81307524, -0.43660507, -0.51048666],
[-0.69864446, 0.37800577, -0.06189097],
[-0.12871675, 0.36555207, 0.6326951 ],
[ 0.13829602, 0.56905323, 0.09383805]], dtype=float32)>,
<tf.Variable 'Dense_1/bias:0' shape=(3,) dtype=float32, numpy=array([-0.02371155, -0.06548308, 0.17505823], dtype=float32)>]
-如果你想让它们成为 numpy.array 形式：
print(saved_model.layers[0].kernel.numpy())
print(saved_model.layers[0].bias.numpy())
-输出：
array([[ 0.44274166, -0.46638554, -0.40543374],
[-0.81307524, -0.43660507, -0.51048666],
[-0.69864446, 0.37800577, -0.06189097],
[-0.12871675, 0.36555207, 0.6326951 ],
[ 0.13829602, 0.56905323, 0.09383805]], dtype=float32)
array([-0.02371155, -0.06548308, 0.17505823], dtype=float32)
在这种情况下，我们保存整个模型。但是如果你只想保存一层的权重，你可以为此创建一个自定义 callback。
2- 使用自定义回调
我通过继承 tensorflow.keras.callbacks.Callback 创建了一个回调。它现在唯一做的就是打印层的权重，您还可以添加代码以使用 pickle 或 numpy.
保存该层的权重
from tensorflow.keras.callbacks import Callback
import pickle
import numpy as np
class CustomCallback(Callback):
def __init__(self, save_path='./logDir', layer_index = 0):
self.save_path = save_path
self.layer_index = layer_index
def on_epoch_end(self, epoch, logs=None):
# access the model weihts
weights_of_first_layer = self.model.layers[self.layer_index].weights
# Do some printing
print(f'\n\nIn the custom callback, Epoch {epoch}: ')
print(f'First layer weights: \n{weights_of_first_layer}')
print('\n\n')
# get weights in the numpy array format
weights = self.model.layers[self.layer_index].kernel.numpy()
biases = self.model.layers[self.layer_index].bias.numpy()
# Now here you can use numpy or pickle to save the weights
#using pickle
#pickle.dump()
# using numpy
# np.save()
-让我们看看这个回调的实际效果：
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
custom_callback = CustomCallback()
model.fit(train_x, train_y,
batch_size=32,
epochs=100,
verbose=1,
callbacks=[custom_callback]
)
-一些fit输出：
Epoch 1/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
In the custom callback, Epoch 0:
First layer weights:
[<tf.Variable 'Dense_1/kernel:0' shape=(5, 3) dtype=float32, numpy=
array([[ 0.270913 , -0.52936906, -0.703977 ],
[-0.9254448 , -0.4501195 , -0.6954986 ],
[-0.84005284, 0.34274203, -0.3004068 ],
[-0.24770388, 0.34638566, 0.43633664],
[ 0.2538888 , 0.5864706 , 0.28424913]], dtype=float32)>, <tf.Variable 'Dense_1/bias:0' shape=(3,) dtype=float32, numpy=array([ 0.02709604, -0.07876156, 0.2504825 ], dtype=float32)>]
Epoch 2/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
In the custom callback, Epoch 1:
First layer weights:
[<tf.Variable 'Dense_1/kernel:0' shape=(5, 3) dtype=float32, numpy=
array([[ 0.27090982, -0.52937293, -0.703984 ],
[-0.92544633, -0.4501213 , -0.6955019 ],
[-0.84005505, 0.3427393 , -0.3004116 ],
[-0.24770552, 0.3463836 , 0.43633294],
[ 0.25389025, 0.5864727 , 0.28425238]], dtype=float32)>, <tf.Variable 'Dense_1/bias:0' shape=(3,) dtype=float32, numpy=array([ 0.02709637, -0.07876115, 0.2504833 ], dtype=float32)>]
Epoch 3/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
一些有用的文档：
我想在 Tensorflow 中编写一个自定义层，我需要在其中保存该层的先前权重（因此对于最后 n 个时期中的每一个）。我会在自定义图层的哪个位置执行此操作？
例如，这是自定义层在 tensorflow 示例中的样子：
class MyDenseLayer(tf.keras.layers.Layer):
def __init__(self, num_outputs):
super(MyDenseLayer, self).__init__()
self.num_outputs = num_outputs
def build(self, input_shape):
self.kernel = self.add_weight("kernel",
shape=[int(input_shape[-1]),
self.num_outputs])
def call(self, input):
return tf.matmul(input, self.kernel)
call() 每个批次调用方法。
在 __init__() 内添加计数器并在 call() 内保存：
class MyDenseLayer(tf.keras.layers.Layer):
def __init__(self, num_outputs):
super(MyDenseLayer, self).__init__()
self.num_outputs = num_outputs
self.counter = 0
def call(self, input):
self.counter += 1
if self.counter % batches_per_epoch == 0:
# add saving here
return tf.matmul(input, self.kernel)
这是我的方法：
1- 使用 ModelCheckpoint 回调
这里我创建了一个前馈神经网络。然后，我使用 tf.keras.callbacks.ModelCheckpoint 在每个时期保存模型。最后，我加载一个保存的模型并访问它的权重。
-首先，让我们创建一个简单的前馈神经网络（当然，您可以使用任何其他层或模型）：
import tensorflow as tf
from tensorflow.keras.layers import Dense
from tensorflow.keras import Sequential
from tensorflow.keras.callbacks import ModelCheckpoint
from sklearn.datasets import make_blobs
from tensorflow.keras.utils import to_categorical
model = Sequential()
model.add(Dense(units=3 , input_dim=5 , activation='relu', name='Dense_1'))
model.add(Dense(units=2 , activation='softmax', name='Dense_2'))
model.summary()
-这里是摘要输出：
Model: "sequential_6"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
Dense_1 (Dense) (None, 3) 18
_________________________________________________________________
Dense_2 (Dense) (None, 2) 8
=================================================================
Total params: 26
Trainable params: 26
Non-trainable params: 0
_________________________________________________________________
-创建用于训练模型的虚拟数据：
train_x , train_y = make_blobs(n_samples=1000, centers=2, n_features=5)
train_y = to_categorical(train_y,2)
-编译并训练模型：
LOG_DIRECTORY = './Whosebug/'
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model_checkpoint = ModelCheckpoint(LOG_DIRECTORY+'weights{epoch:03d}.h5',
save_freq='epoch',
verbose=1)
model.fit(train_x, train_y,
batch_size=32,
epochs=100,
verbose=1,
callbacks=[model_checkpoint]
)
-让我们看看一些 fit 输出：
Epoch 1/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
Epoch 00001: saving model to ./Whosebug/weights001.h5
Epoch 2/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
Epoch 00002: saving model to ./Whosebug/weights002.h5
Epoch 3/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
Epoch 00003: saving model to ./Whosebug/weights003.h5
Epoch 4/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
Epoch 00004: saving model to ./Whosebug/weights004.h5
正如您在上面的输出中看到的，模型在每个时期后使用 model_checkpoint = ModelCheckpoint ....
-加载保存的模型：
from tensorflow.keras.models import load_model
saved_model =load_model(LOG_DIRECTORY+'weights097.h5')
-打印层的重量：
print(saved_model.layers[0].weights)
在您的情况下，使用所需图层的索引。
-输出：
输出是 tf.Variable 个权重和偏差
[<tf.Variable 'Dense_1/kernel:0' shape=(5, 3) dtype=float32, numpy=
array([[ 0.44274166, -0.46638554, -0.40543374],
[-0.81307524, -0.43660507, -0.51048666],
[-0.69864446, 0.37800577, -0.06189097],
[-0.12871675, 0.36555207, 0.6326951 ],
[ 0.13829602, 0.56905323, 0.09383805]], dtype=float32)>,
<tf.Variable 'Dense_1/bias:0' shape=(3,) dtype=float32, numpy=array([-0.02371155, -0.06548308, 0.17505823], dtype=float32)>]
-如果你想让它们成为 numpy.array 形式：
print(saved_model.layers[0].kernel.numpy())
print(saved_model.layers[0].bias.numpy())
-输出：
array([[ 0.44274166, -0.46638554, -0.40543374],
[-0.81307524, -0.43660507, -0.51048666],
[-0.69864446, 0.37800577, -0.06189097],
[-0.12871675, 0.36555207, 0.6326951 ],
[ 0.13829602, 0.56905323, 0.09383805]], dtype=float32)
array([-0.02371155, -0.06548308, 0.17505823], dtype=float32)
在这种情况下，我们保存整个模型。但是如果你只想保存一层的权重，你可以为此创建一个自定义 callback。
2- 使用自定义回调
我通过继承 tensorflow.keras.callbacks.Callback 创建了一个回调。它现在唯一做的就是打印层的权重，您还可以添加代码以使用 pickle 或 numpy.
from tensorflow.keras.callbacks import Callback
import pickle
import numpy as np
class CustomCallback(Callback):
def __init__(self, save_path='./logDir', layer_index = 0):
self.save_path = save_path
self.layer_index = layer_index
def on_epoch_end(self, epoch, logs=None):
# access the model weihts
weights_of_first_layer = self.model.layers[self.layer_index].weights
# Do some printing
print(f'\n\nIn the custom callback, Epoch {epoch}: ')
print(f'First layer weights: \n{weights_of_first_layer}')
print('\n\n')
# get weights in the numpy array format
weights = self.model.layers[self.layer_index].kernel.numpy()
biases = self.model.layers[self.layer_index].bias.numpy()
# Now here you can use numpy or pickle to save the weights
#using pickle
#pickle.dump()
# using numpy
# np.save()
-让我们看看这个回调的实际效果：
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
custom_callback = CustomCallback()
model.fit(train_x, train_y,
batch_size=32,
epochs=100,
verbose=1,
callbacks=[custom_callback]
)
-一些fit输出：
Epoch 1/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
In the custom callback, Epoch 0:
First layer weights:
[<tf.Variable 'Dense_1/kernel:0' shape=(5, 3) dtype=float32, numpy=
array([[ 0.270913 , -0.52936906, -0.703977 ],
[-0.9254448 , -0.4501195 , -0.6954986 ],
[-0.84005284, 0.34274203, -0.3004068 ],
[-0.24770388, 0.34638566, 0.43633664],
[ 0.2538888 , 0.5864706 , 0.28424913]], dtype=float32)>, <tf.Variable 'Dense_1/bias:0' shape=(3,) dtype=float32, numpy=array([ 0.02709604, -0.07876156, 0.2504825 ], dtype=float32)>]
Epoch 2/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
In the custom callback, Epoch 1:
First layer weights:
[<tf.Variable 'Dense_1/kernel:0' shape=(5, 3) dtype=float32, numpy=
array([[ 0.27090982, -0.52937293, -0.703984 ],
[-0.92544633, -0.4501213 , -0.6955019 ],
[-0.84005505, 0.3427393 , -0.3004116 ],
[-0.24770552, 0.3463836 , 0.43633294],
[ 0.25389025, 0.5864727 , 0.28425238]], dtype=float32)>, <tf.Variable 'Dense_1/bias:0' shape=(3,) dtype=float32, numpy=array([ 0.02709637, -0.07876115, 0.2504833 ], dtype=float32)>]
Epoch 3/100
32/32 [==============================] - 0s 1ms/step - loss: 0.0000e+00 - accuracy: 1.0000
一些有用的文档：