如何使用 Keras 实现自定义的基于 sobel-filter 的损失函数

Question

我是 DL 和 Keras 的新手，目前我正在尝试在 Keras 中实现基于 sobel-filter 的自定义损失函数。

想法是计算 sobel 滤波预测和 sobel 滤波地面真值图像的均方损失。

到目前为止，我的自定义损失函数如下所示：

from scipy import ndimage

def mse_sobel(y_true, y_pred):

    for i in range (0, y_true.shape[0]):
        dx_true = ndimage.sobel(y_true[i,:,:,:], 1)
        dy_true = ndimage.sobel(y_true[i,:,:,:], 2)
        mag_true[i,:,:,:] = np.hypot(dx_true, dy_true)
        mag_true[i,:,:,:] *= 1.0 / np.max(mag_true[i,:,:,:])

        dx_pred = ndimage.sobel(y_pred[i,:,:,:], 1)
        dy_pred = ndimage.sobel(y_pred[i,:,:,:], 2)
        mag_pred[i,:,:,:] = np.hypot(dx_pred, dy_pred)
        mag_pred[i,:,:,:] *= 1.0 / np.max(mag_pred[i,:,:,:])

    return(K.mean(K.square(mag_pred - mag_true), axis=-1))

使用这个损失函数会导致这个错误：

in mse_sobel
for i in range (0, y_true.shape[0]):
TypeError: __index__ returned non-int (type NoneType)

使用我发现的调试器，y_true.shape 仅 returns None - 很好。但是，当我将 y_true.shape 替换为 1 使其看起来像这样 for i in range (0,1): 时，会发生另一个错误：

in sobel
axis = _ni_support._check_axis(axis, input.ndim)

in _check_axis
raise ValueError('invalid axis')
ValueError: invalid axis

这里，我不太清楚为什么轴好像是无效的？

谁能帮我弄清楚如何实现那个损失函数？非常感谢您的帮助！

Answer 1

损失必须使用张量运算，使用keras后端，或tensorflow/theano/cntk函数。这是保持反向传播的唯一方法。使用 numpy，scipy 等打破了图形。

让我们导入keras后端：

import keras.backend as K

定义过滤器：

#this contains both X and Y sobel filters in the format (3,3,1,2)
#size is 3 x 3, it considers 1 input channel and has two output channels: X and Y
sobelFilter = K.variable([[[[1.,  1.]], [[0.,  2.]],[[-1.,  1.]]],
                      [[[2.,  0.]], [[0.,  0.]],[[-2.,  0.]]],
                      [[[1., -1.]], [[0., -2.]],[[-1., -1.]]]])

这里是一个为每个输入通道重复过滤器的函数，以防您的图像是 RGB 或具有多个通道。这只会为每个输入通道复制 sobel 滤波器：(3,3,inputChannels, 2):

def expandedSobel(inputTensor):

    #this considers data_format = 'channels_last'
    inputChannels = K.reshape(K.ones_like(inputTensor[0,0,0,:]),(1,1,-1,1))
    #if you're using 'channels_first', use inputTensor[0,:,0,0] above

    return sobelFilter * inputChannels

这是损失函数：

def sobelLoss(yTrue,yPred):

    #get the sobel filter repeated for each input channel
    filt = expandedSobel(yTrue)

    #calculate the sobel filters for yTrue and yPred
    #this generates twice the number of input channels 
    #a X and Y channel for each input channel
    sobelTrue = K.depthwise_conv2d(yTrue,filt)
    sobelPred = K.depthwise_conv2d(yPred,filt)

    #now you just apply the mse:
    return K.mean(K.square(sobelTrue - sobelPred))

在模型中应用此损失：

model.compile(loss=sobelLoss, optimizer = ....)

我的经验表明，计算统一的 sobel 滤波器 sqrt(X² + Y²) 会带来糟糕的结果，并且生成的图像听起来像棋盘。但如果你确实想要它：

def squareSobelLoss(yTrue,yPred):

    #same beginning as the other loss
    filt = expandedSobel(yTrue)
    squareSobelTrue = K.square(K.depthwise_conv2d(yTrue,filt))
    squareSobelPred = K.square(K.depthwise_conv2d(yPred,filt))

    #here, since we've got 6 output channels (for an RGB image)
    #let's reorganize in order to easily sum X² and Y²: change (h,w,6) to (h,w,3,2)
    #caution: this method of reshaping only works in tensorflow
    #if you do need this in other backends, let me know
    newShape = K.shape(squareSobelTrue)
    newShape = K.concatenate([newShape[:-1],
                              newShape[-1:]//2,
                              K.variable([2],dtype='int32')])

    #sum the last axis (the one that is 2 above, representing X² and Y²)                      
    squareSobelTrue = K.sum(K.reshape(squareSobelTrue,newShape),axis=-1)
    squareSobelPred = K.sum(K.reshape(squareSobelPred,newShape),axis=-1)

    #since both previous values are already squared, maybe we shouldn't square them again? 
    #but you can apply the K.sqrt() in both, and then make the difference, 
    #and then another square, it's up to you...    
    return K.mean(K.abs(squareSobelTrue - squareSobelPred))

如何使用 Keras 实现自定义的基于 sobel-filter 的损失函数

How to implement custom sobel-filter-based loss function using Keras

python

scipy

sobel

keras