Keras 中的正则化：如何控制最后一层零权重的最大数量？

Question

我有一个神经网络，最后一层输出一个大小为 N (N=8) 的向量。当我进行多标签分类时，我发现大部分输出向量元素都为零，最多有两个元素等于 1。例如 y_pred == [1, 0, 0, 0, 0, 0, 0, 1].

我想告诉我的网络，即至少 N-2 个输出权重等于 0。

我现在的模型如下：

ResNet18, preprocess_input = Classifiers.get('resnet18')
resnet = ResNet18((im_size, im_size, 3), weights='imagenet', include_top=False)
headModel = keras.layers.pooling.AveragePooling2D(pool_size=(3,3))(resnet.output)
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(256, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
    
# 'sigmoid' parameter indicating that we’ll be performing multi-label classification.
headModel = Dense(8, activation="sigmoid")(headModel)

我正在考虑将正则化器 my_reg 添加到我的最后一个 Dense 层，这样类似于

headModel = Dense(8, activation="sigmoid", kernel_regularizer=my_reg)(headModel)

我没有使用 Keras 中的正则化器以及如何操纵权重的经验。

Answer 1

您可以创建一个自定义函数作为您的激活函数。更具体地说，将两个最小概率设置为零。

def custom_func(x):
    second_smallest = tf.sort(tf.squeeze(x))[1]
    x = tf.where(second_smallest >= x, tf.zeros_like(x), x)
    return x

import numpy as np
import tensorflow as tf

inp = tf.keras.Input(shape=(224, 224, 3))
base = tf.keras.applications.MobileNetV2(include_top=False, 
                                         input_shape=(224, 224, 3))(inp)
gap = tf.keras.layers.GlobalAveragePooling2D()(base)
out = tf.keras.layers.Dense(8, activation='sigmoid')(gap)
custom_function = tf.keras.layers.Lambda(custom_func)(out)

model = tf.keras.Model(inp, custom_function)

model(np.random.rand(1, 224, 224, 3).astype(np.float32))

<tf.Tensor: shape=(1, 8), dtype=float32, numpy=
array([[0.36225533, 0.66996753, 0.9467776 , 0.        , 0.6429986 ,
        0.9498544 , 0.        , 0.6883256 ]], dtype=float32)>

您也可以让它接受这样的参数：

import numpy as np
import tensorflow as tf


def custom_func(inputs, n_to_zero):
    second_smallest = tf.sort(tf.squeeze(inputs))[n_to_zero - 1]
    out = tf.where(second_smallest >= inputs, tf.zeros_like(inputs), inputs)
    return out


inp = tf.keras.Input(shape=(224, 224, 3))
base = tf.keras.applications.MobileNetV2(include_top=False, 
                                         input_shape=(224, 224, 3))(inp)
gap = tf.keras.layers.GlobalAveragePooling2D()(base)
out = tf.keras.layers.Dense(8, activation='sigmoid')(gap)
custom_function = tf.keras.layers.Lambda(
    lambda x: custom_func(inputs=x, n_to_zero=4)
                                        )(out)

model = tf.keras.Model(inp, custom_function)

model(np.random.rand(1, 224, 224, 3).astype(np.float32))

<tf.Tensor: shape=(1, 8), dtype=float32, numpy=
array([[0.8537902, 0.       , 0.       , 0.       , 0.7386258, 0.       ,
        0.0948523, 0.7973974]], dtype=float32)>

Keras 中的正则化：如何控制最后一层零权重的最大数量？

Regularization in Keras: How can I control the maximum number of zero weights on the last layer?

python

constraints

deep-learning

keras

tensorflow