TensorFlow、Keras：替换预训练模型中的激活层

Question

我正在尝试在预训练的 TF 模型 EfficientNetB0 中用 relu 激活替换 swish 激活。 EfficientNetB0 在 Conv2D 和激活层中使用 swish 激活。这 SO post is very similar to what I'm looking for. I also found 适用于没有跳过连接的模型。下面是代码：

import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import ReLU

def replace_swish_with_relu(model):
    '''
    Modify passed model by replacing swish activation with relu
    '''
    for layer in tuple(model.layers):
        layer_type = type(layer).__name__
        if hasattr(layer, 'activation') and layer.activation.__name__ == 'swish':
            print(layer_type, layer.activation.__name__)
            if layer_type == "Conv2D":
                # conv layer with swish activation.
                # Do something
                layer.activation = ReLU() # This didn't work
            else:
                # activation layer
                # Do something
                layer = tf.keras.layers.Activation('relu', name=layer.name + "_relu") # This didn't work
    return model

# load pretrained efficientNet
model = tf.keras.applications.EfficientNetB0(
    include_top=True, weights='imagenet', input_tensor=None,
    input_shape=(224, 224, 3), pooling=None, classes=1000,
    classifier_activation='softmax')

# convert swish activation to relu activation
model = replace_swish_with_relu(model)
model.save("efficientNet-relu")

如何修改replace_swish_with_relu以在传递的模型中用relu替换swish激活？

感谢任何pointers/help。

Answer 1

试试这个：

def replace_swish_with_relu(model):
    '''
    Modify passed model by replacing swish activation with relu
    '''
    for i,layer in enumerate(tuple(model.layers)):
        layer_type = type(layer).__name__
        if hasattr(layer, 'activation') and layer.activation.__name__ == 'swish':
            print(layer_type, layer.activation.__name__)
            if layer_type == "Conv2D":
                # conv layer with swish activation.
                # Do something
                model.layers[i] = ReLU() # This didn't work
            else:
                # activation layer
                # Do something
                model.layers[i] = tf.keras.layers.Activation('relu', name=layer.name + "_relu") # This didn't work
    return model

Answer 2

layer.activation指向tf.keras.activations.swish函数地址。我们可以修改它指向tf.keras.activations.relu。下面是修改后的，replace_swish_with_relu:

def replace_swish_with_relu(model):
    '''
    Modify passed model by replacing swish activation with relu
    '''
    for layer in tuple(model.layers):
        layer_type = type(layer).__name__
        if hasattr(layer, 'activation') and layer.activation.__name__ == 'swish':
            print(layer_type, layer.activation.__name__)
            if layer_type == "Conv2D":
                # conv layer with swish activation
                layer.activation = tf.keras.activations.relu
            else:
                # activation layer
                layer.activation = tf.keras.activations.relu
    return model

注意：如果您要修改激活函数，则需要重新训练模型以使用新的激活函数。 Related.

TensorFlow、Keras：替换预训练模型中的激活层

TensorFlow, Keras: Replace Activation layer in pretrained model

python

tensorflow

keras-2

tf.keras

tensorflow2.0