具有共享权重的 Keras 密集模型

Question

我想在 keras 中创建一个具有共享权重的密集模型。我的方法如下所述。我有一个名为 dense 的密集网络。 dense_x、dense_y 和 dense_z 应该共享权重（即使用 dense）。然后将这三个网络的输出连接起来并馈入另一个密集网络。

然而，不知何故这种方法不起作用。如何正确使用共享权重？

num_nodes = 310
input_tensor_x = Input(shape=(310,))
input_tensor_y = Input(shape=(310,))
input_tensor_z = Input(shape=(310,))

dense = Dense(num_nodes)
dense = BatchNormalization()(dense)
dense = Activation('relu')(dense)
dense = Dropout(0.4)(dense)

dense = Dense(num_nodes // 2)
dense = BatchNormalization()(dense)
dense = Activation('relu')(dense)
dense = Dropout(0.4)(dense)

dense = Dense(num_nodes // 4)
dense = BatchNormalization()(dense)
dense = Activation('relu')(dense)
dense = Dropout(0.4)(dense)

dense_x = dense(input_tensor_x) #<=== shared above dense network
dense_y = dense(input_tensor_y) #<=== shared above dense network
dense_z = dense(input_tensor_z) #<=== shared above dense network
merge_layer = tf.keras.layers.Concatenate()([dense_x, dense_y, dense_z])

merged_nodes = 3*(num_nodes // 4) // 2
dense2 = Dense(merged_nodes)(merge_layer)
dense2 = BatchNormalization()(dense2)
dense2 = Activation('relu')(dense2)
dense2 = Dropout(0.4)(dense2)

dense2 = Dense(merged_nodes // 2)(dense2)
dense2 = BatchNormalization()(dense2)
dense2 = Activation('relu')(dense2)
dense2 = Dropout(0.4)(dense2)

output_tensor = Dense(3, activation='softmax')(dense2)

fcnn_model = Model(inputs=[input_tensor_x, input_tensor_y, input_tensor_z], outputs=output_tensor)
opt = Adam(lr=learning_rate)
fcnn_model.compile(loss='categorical_crossentropy',
                  optimizer=opt, metrics=['accuracy', tf.keras.metrics.AUC()])

Answer 1

最简单的方法是在顶部初始化共享层，然后将下面的层传递给它们。注意不要覆盖它们。

我初始化了一个 Dense 层和一个 BatchNormalization 因为它们是唯一具有可训练权重的层

num_nodes = 310
input_tensor_x = Input(shape=(310,))
input_tensor_y = Input(shape=(310,))
input_tensor_z = Input(shape=(310,))

def shared_dense(inp):
    
    dense = Dense(num_nodes)(inp)
    dense = BatchNormalization()(dense)
    dense = Activation('relu')(dense)
    dense = Dropout(0.4)(dense)

    dense = Dense(num_nodes // 2)(dense)
    dense = BatchNormalization()(dense)
    dense = Activation('relu')(dense)
    dense = Dropout(0.4)(dense)

    dense = Dense(num_nodes // 4)(dense)
    dense = BatchNormalization()(dense)
    dense = Activation('relu')(dense)
    dense = Dropout(0.4)(dense)

    return Model(inp, dense, name='shared_dense')

shared_dense = shared_dense(Input(shape=(310,)))
dense_x = shared_dense(input_tensor_x) #<=== shared 
dense_y = shared_dense(input_tensor_y) #<=== shared 
dense_z = shared_dense(input_tensor_z) #<=== shared 

merge_layer = Concatenate()([dense_x, dense_y, dense_z])

merged_nodes = 3*(num_nodes // 4) // 2
dense2 = Dense(merged_nodes)(merge_layer)
dense2 = BatchNormalization()(dense2)
dense2 = Activation('relu')(dense2)
dense2 = Dropout(0.4)(dense2)

dense2 = Dense(merged_nodes // 2)(dense2)
dense2 = BatchNormalization()(dense2)
dense2 = Activation('relu')(dense2)
dense2 = Dropout(0.4)(dense2)

output_tensor = Dense(3, activation='softmax')(dense2)

fcnn_model = Model(inputs=[input_tensor_x, input_tensor_y, input_tensor_z], 
                   outputs=output_tensor)
fcnn_model.compile(loss='categorical_crossentropy',
                  optimizer=Adam(lr=1e-3), 
                   metrics=['accuracy', tf.keras.metrics.AUC()])

具有共享权重的 Keras 密集模型

Keras Dense models with shared weights

python

neural-network

keras

tensorflow

tf.keras