以自定义张量作为变量的 TensorFlow 2.0 Keras 层
TensorFlow 2.0 Keras layers with custom tensors as variables
在 TF 1.x 中,可以使用自定义变量构建图层。这是一个例子:
import numpy as np
import tensorflow as tf
def make_custom_getter(custom_variables):
def custom_getter(getter, name, **kwargs):
if name in custom_variables:
variable = custom_variables[name]
else:
variable = getter(name, **kwargs)
return variable
return custom_getter
# Make a custom getter for the dense layer variables.
# Note: custom variables can result from arbitrary computation;
# for the sake of this example, we make them just constant tensors.
custom_variables = {
"model/dense/kernel": tf.constant(
np.random.rand(784, 64), name="custom_kernel", dtype=tf.float32),
"model/dense/bias": tf.constant(
np.random.rand(64), name="custom_bias", dtype=tf.float32),
}
custom_getter = make_custom_getter(custom_variables)
# Compute hiddens using a dense layer with custom variables.
x = tf.random.normal(shape=(1, 784), name="inputs")
with tf.variable_scope("model", custom_getter=custom_getter):
Layer = tf.layers.Dense(64)
hiddens = Layer(x)
print(Layer.variables)
构造的致密层的打印变量将是我们在custom_variables
dict中指定的自定义张量:
[<tf.Tensor 'custom_kernel:0' shape=(784, 64) dtype=float32>, <tf.Tensor 'custom_bias:0' shape=(64,) dtype=float32>]
这允许我们创建 layers/models 直接使用 custom_variables
中提供的张量作为它们的权重,这样我们就可以进一步区分 layers/models 相对于任何张量的输出custom_variables
可能依赖于(对于在 modulating sub-nets, parameter generation, meta-learning 等中实现功能特别有用)。
变量作用域使使用自定义 getter 的作用域内嵌套所有图形构建变得容易,并在提供的张量之上构建模型作为它们的参数。由于在 TF 2.0 中不再建议使用会话和变量作用域(并且所有这些低级内容都移到了 tf.compat.v1
),因此 最佳实践 是什么来实现以上使用 Keras 和 TF 2.0?
(相关 issue on GitHub。)
根据下面的评论回答
假设你有:
kernel = createTheKernelVarBasedOnWhatYouWant() #shape (784, 64)
bias = createTheBiasVarBasedOnWhatYouWant() #shape (64,)
从 Dense
:
复制代码制作一个简单的函数
def custom_dense(x):
inputs, kernel, bias = x
outputs = K.dot(inputs, kernel)
outputs = K.bias_add(outputs, bias, data_format='channels_last')
return outputs
在Lambda
层使用函数:
layer = Lambda(custom_dense)
hiddens = layer([x, kernel, bias])
Warning: kernel
and bias
must be produced from a Keras layer, or come from an kernel = Input(tensor=the_kernel_var)
and bias = Input(tensor=bias_var)
如果上面的警告对您不利,您可以随时使用 kernel
和 bias
“从外部”,例如:
def custom_dense(inputs):
outputs = K.dot(inputs, kernel) #where kernel is not part of the arguments anymore
outputs = K.bias_add(outputs, bias, data_format='channels_last')
return outputs
layer = Lambda(custom_dense)
hiddens = layer(x)
最后一个选项使 save/load 模型变得有点复杂。
旧答案
您可能应该使用 Keras Dense 层并以标准方式设置其权重:
layer = tf.keras.layers.Dense(64, name='the_layer')
layer.set_weights([np.random.rand(784, 64), np.random.rand(64)])
如果你需要这些权重是不可训练的,在编译你设置的keras模型之前:
model.get_layer('the_layer').trainable=False
如果您想直接访问变量作为张量,它们是:
kernel = layer.kernel
bias = layer.bias
还有很多其他选择,但这取决于您的确切意图,这在您的问题中并不清楚。
不完全确定我是否正确理解了你的问题,但在我看来,应该可以通过 custom layers and keras functional api.
的组合来做你想做的事
自定义层允许您以与 Keras 兼容的方式构建您想要的任何层,例如:
class MyDenseLayer(tf.keras.layers.Layer):
def __init__(self, num_outputs):
super(MyDenseLayer, self).__init__()
self.num_outputs = num_outputs
def build(self, input_shape):
self.kernel = self.add_weight("kernel",
shape=[int(input_shape[-1]),
self.num_outputs],
initializer='normal')
self.bias = self.add_weight("bias",
shape=[self.num_outputs,],
initializer='normal')
def call(self, inputs):
return tf.matmul(inputs, self.kernel) + self.bias
并且函数 api 允许您访问所述层的输出并重新使用它们:
inputs = keras.Input(shape=(784,), name='img')
x1 = MyDenseLayer(64, activation='relu')(inputs)
x2 = MyDenseLayer(64, activation='relu')(x1)
outputs = MyDenseLayer(10, activation='softmax')(x2)
model = keras.Model(inputs=inputs, outputs=outputs, name='mnist_model')
此处x1
和x2
可以连接到其他子网。
下面是一个通用解决方案,适用于 TF2 中的任意 Keras 模型。
首先,我们需要定义一个辅助函数 canonical_variable_name
和一个具有以下签名的上下文管理器 custom_make_variable
(参见 meta-blocks library 中的实现)。
def canonical_variable_name(variable_name: str, outer_scope: str):
"""Returns the canonical variable name: `outer_scope/.../name`."""
# ...
@contextlib.contextmanager
def custom_make_variable(
canonical_custom_variables: Dict[str, tf.Tensor], outer_scope: str
):
"""A context manager that overrides `make_variable` with a custom function.
When building layers, Keras uses `make_variable` function to create weights
(kernels and biases for each layer). This function wraps `make_variable` with
a closure that infers the canonical name of the variable being created (of the
form `outer_scope/.../var_name`) and looks it up in the `custom_variables` dict
that maps canonical names to tensors. The function adheres the following logic:
* If there is a match, it does a few checks (shape, dtype, etc.) and returns
the found tensor instead of creating a new variable.
* If there is a match but checks fail, it throws an exception.
* If there are no matching `custom_variables`, it calls the original
`make_variable` utility function and returns a newly created variable.
"""
# ...
使用这些函数,我们可以使用自定义张量作为变量创建任意 Keras 模型:
import numpy as np
import tensorflow as tf
canonical_custom_variables = {
"model/dense/kernel": tf.constant(
np.random.rand(784, 64), name="custom_kernel", dtype=tf.float32),
"model/dense/bias": tf.constant(
np.random.rand(64), name="custom_bias", dtype=tf.float32),
}
# Compute hiddens using a dense layer with custom variables.
x = tf.random.normal(shape=(1, 784), name="inputs")
with custom_make_variable(canonical_custom_variables, outer_scope="model"):
Layer = tf.layers.Dense(64)
hiddens = Layer(x)
print(Layer.variables)
在 TF 1.x 中,可以使用自定义变量构建图层。这是一个例子:
import numpy as np
import tensorflow as tf
def make_custom_getter(custom_variables):
def custom_getter(getter, name, **kwargs):
if name in custom_variables:
variable = custom_variables[name]
else:
variable = getter(name, **kwargs)
return variable
return custom_getter
# Make a custom getter for the dense layer variables.
# Note: custom variables can result from arbitrary computation;
# for the sake of this example, we make them just constant tensors.
custom_variables = {
"model/dense/kernel": tf.constant(
np.random.rand(784, 64), name="custom_kernel", dtype=tf.float32),
"model/dense/bias": tf.constant(
np.random.rand(64), name="custom_bias", dtype=tf.float32),
}
custom_getter = make_custom_getter(custom_variables)
# Compute hiddens using a dense layer with custom variables.
x = tf.random.normal(shape=(1, 784), name="inputs")
with tf.variable_scope("model", custom_getter=custom_getter):
Layer = tf.layers.Dense(64)
hiddens = Layer(x)
print(Layer.variables)
构造的致密层的打印变量将是我们在custom_variables
dict中指定的自定义张量:
[<tf.Tensor 'custom_kernel:0' shape=(784, 64) dtype=float32>, <tf.Tensor 'custom_bias:0' shape=(64,) dtype=float32>]
这允许我们创建 layers/models 直接使用 custom_variables
中提供的张量作为它们的权重,这样我们就可以进一步区分 layers/models 相对于任何张量的输出custom_variables
可能依赖于(对于在 modulating sub-nets, parameter generation, meta-learning 等中实现功能特别有用)。
变量作用域使使用自定义 getter 的作用域内嵌套所有图形构建变得容易,并在提供的张量之上构建模型作为它们的参数。由于在 TF 2.0 中不再建议使用会话和变量作用域(并且所有这些低级内容都移到了 tf.compat.v1
),因此 最佳实践 是什么来实现以上使用 Keras 和 TF 2.0?
(相关 issue on GitHub。)
根据下面的评论回答
假设你有:
kernel = createTheKernelVarBasedOnWhatYouWant() #shape (784, 64)
bias = createTheBiasVarBasedOnWhatYouWant() #shape (64,)
从 Dense
:
def custom_dense(x):
inputs, kernel, bias = x
outputs = K.dot(inputs, kernel)
outputs = K.bias_add(outputs, bias, data_format='channels_last')
return outputs
在Lambda
层使用函数:
layer = Lambda(custom_dense)
hiddens = layer([x, kernel, bias])
Warning:
kernel
andbias
must be produced from a Keras layer, or come from ankernel = Input(tensor=the_kernel_var)
andbias = Input(tensor=bias_var)
如果上面的警告对您不利,您可以随时使用 kernel
和 bias
“从外部”,例如:
def custom_dense(inputs):
outputs = K.dot(inputs, kernel) #where kernel is not part of the arguments anymore
outputs = K.bias_add(outputs, bias, data_format='channels_last')
return outputs
layer = Lambda(custom_dense)
hiddens = layer(x)
最后一个选项使 save/load 模型变得有点复杂。
旧答案
您可能应该使用 Keras Dense 层并以标准方式设置其权重:
layer = tf.keras.layers.Dense(64, name='the_layer')
layer.set_weights([np.random.rand(784, 64), np.random.rand(64)])
如果你需要这些权重是不可训练的,在编译你设置的keras模型之前:
model.get_layer('the_layer').trainable=False
如果您想直接访问变量作为张量,它们是:
kernel = layer.kernel
bias = layer.bias
还有很多其他选择,但这取决于您的确切意图,这在您的问题中并不清楚。
不完全确定我是否正确理解了你的问题,但在我看来,应该可以通过 custom layers and keras functional api.
的组合来做你想做的事自定义层允许您以与 Keras 兼容的方式构建您想要的任何层,例如:
class MyDenseLayer(tf.keras.layers.Layer):
def __init__(self, num_outputs):
super(MyDenseLayer, self).__init__()
self.num_outputs = num_outputs
def build(self, input_shape):
self.kernel = self.add_weight("kernel",
shape=[int(input_shape[-1]),
self.num_outputs],
initializer='normal')
self.bias = self.add_weight("bias",
shape=[self.num_outputs,],
initializer='normal')
def call(self, inputs):
return tf.matmul(inputs, self.kernel) + self.bias
并且函数 api 允许您访问所述层的输出并重新使用它们:
inputs = keras.Input(shape=(784,), name='img')
x1 = MyDenseLayer(64, activation='relu')(inputs)
x2 = MyDenseLayer(64, activation='relu')(x1)
outputs = MyDenseLayer(10, activation='softmax')(x2)
model = keras.Model(inputs=inputs, outputs=outputs, name='mnist_model')
此处x1
和x2
可以连接到其他子网。
下面是一个通用解决方案,适用于 TF2 中的任意 Keras 模型。
首先,我们需要定义一个辅助函数 canonical_variable_name
和一个具有以下签名的上下文管理器 custom_make_variable
(参见 meta-blocks library 中的实现)。
def canonical_variable_name(variable_name: str, outer_scope: str):
"""Returns the canonical variable name: `outer_scope/.../name`."""
# ...
@contextlib.contextmanager
def custom_make_variable(
canonical_custom_variables: Dict[str, tf.Tensor], outer_scope: str
):
"""A context manager that overrides `make_variable` with a custom function.
When building layers, Keras uses `make_variable` function to create weights
(kernels and biases for each layer). This function wraps `make_variable` with
a closure that infers the canonical name of the variable being created (of the
form `outer_scope/.../var_name`) and looks it up in the `custom_variables` dict
that maps canonical names to tensors. The function adheres the following logic:
* If there is a match, it does a few checks (shape, dtype, etc.) and returns
the found tensor instead of creating a new variable.
* If there is a match but checks fail, it throws an exception.
* If there are no matching `custom_variables`, it calls the original
`make_variable` utility function and returns a newly created variable.
"""
# ...
使用这些函数,我们可以使用自定义张量作为变量创建任意 Keras 模型:
import numpy as np
import tensorflow as tf
canonical_custom_variables = {
"model/dense/kernel": tf.constant(
np.random.rand(784, 64), name="custom_kernel", dtype=tf.float32),
"model/dense/bias": tf.constant(
np.random.rand(64), name="custom_bias", dtype=tf.float32),
}
# Compute hiddens using a dense layer with custom variables.
x = tf.random.normal(shape=(1, 784), name="inputs")
with custom_make_variable(canonical_custom_variables, outer_scope="model"):
Layer = tf.layers.Dense(64)
hiddens = Layer(x)
print(Layer.variables)