具有可训练标量的自定义 Keras 层

Question

我正在（尝试）编写一个自定义 Keras 层，它实现了以下组件：

x -> a x + bReLU(x)

具有 a 和 b 可训练权重。以下是我到目前为止所做的尝试：

class Custom_ReLU(tf.keras.layers.Layer):

    def __init__(self, units=d):
        super(Custom_ReLU, self).__init__()
        self.units = units

    def build(self, input_shape):
        self.a1 = self.add_weight(shape=[1],
                                initializer = 'random_uniform',
                                trainable=True)
        self.a2 = self.add_weight(shape=[1],
                                initializer = 'random_uniform',
                                trainable=True)

    def call(self,inputs):
        return self.a1*inputs + self.a2*(tf.nn.relu(inputs))

但是，得到错误。我认为问题在于我不知道如何定义可训练 "scalars"...我的想法是否正确以及如何做到这一点？

Edit/Additions:

以下是我尝试构建我的普通前馈架构的方式，其中 ReLU 被替换为 "Custom_ReLU":

# Build Vanilla Network
inputs_ffNN = tf.keras.Input(shape=(d,))
x_ffNN = fullyConnected_Dense(d)(inputs_ffNN)
for i in range(Depth):
    x_HTC = Custom_ReLU(x_ffNN)
    x_ffNN = fullyConnected_Dense(d)(x_ffNN)
outputs_ffNN = fullyConnected_Dense(D)(x_ffNN)
ffNN = tf.keras.Model(inputs_ffNN, outputs_ffNN)

这里是错误的片段：

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-27-8bf6fc4ae89d> in <module>
      7     #x_HTC = tf.nn.relu(x_HTC)
      8     x_HTC = BounceLU(x_HTC)
----> 9     x_HTC = HTC(d)(x_HTC)
     10 outputs_HTC = HTC(D)(x_HTC)
     11 ffNN_HTC = tf.keras.Model(inputs_HTC, outputs_HTC)

~/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    816         # Eager execution on data tensors.
    817         with backend.name_scope(self._name_scope()):
--> 818           self._maybe_build(inputs)
    819           cast_inputs = self._maybe_cast_inputs(inputs)
    820           with base_layer_utils.autocast_context_manager(

~/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in _maybe_build(self, inputs)
   2114         # operations.
   2115         with tf_utils.maybe_init_scope(self):
-> 2116           self.build(input_shapes)
   2117       # We must set self.built since user defined build functions are not
   2118       # constrained to set self.built.

<ipython-input-5-21623825ed35> in build(self, input_shape)
      5 
      6     def build(self, input_shape):
----> 7         self.w = self.add_weight(shape=(input_shape[-1], self.units),
      8                                initializer='random_normal',
      9                                trainable=False)

TypeError: 'NoneType' object is not subscriptable

Answer 1

我使用你的层没问题：

class Custom_ReLU(tf.keras.layers.Layer):

    def __init__(self):
        super(Custom_ReLU, self).__init__()

        self.a1 = self.add_weight(shape=[1],
                                initializer = 'random_uniform',
                                trainable=True)
        self.a2 = self.add_weight(shape=[1],
                                initializer = 'random_uniform',
                                trainable=True)

    def call(self,inputs):
        return self.a1*inputs + self.a2*(tf.nn.relu(inputs))

用法：

d = 5
inputs_ffNN = tf.keras.Input(shape=(d,))
x_ffNN = tf.keras.layers.Dense(10)(inputs_ffNN)
x_HTC = Custom_ReLU()(x_ffNN)
outputs_ffNN = tf.keras.layers.Dense(1)(x_HTC)

ffNN = tf.keras.Model(inputs_ffNN, outputs_ffNN)
ffNN.compile('adam', 'mse')

ffNN.fit(np.random.uniform(0,1, (10,5)), np.random.uniform(0,1, 10), epochs=10)

这里是完整的例子：https://colab.research.google.com/drive/1n4jIsY3qEDvtobofQaUPO3ysUW9bQWjs?usp=sharing

具有可训练标量的自定义 Keras 层

Custom Keras Layer with Trainable Scalars

python

layer

neural-network

keras

tensorflow