Keras Custom Loss Function InvalidArgumentError: In[1] is not a matrix. Instead it has shape []

Question

我正在尝试使用 Spearman 等级相关系数来编写自定义损失函数。我想计算每对 y_true 和 y_pred 样本之间的 Spearman 秩相关系数（每个样本是一个包含 8 个元素的数组；例如 [1 2 3 4 5 6 7 8] 和 [3 2 1 4 5 8 6 7]).

我已按照此答案的指示进行操作 (How to compute Spearman correlation in Tensorflow) and Keras documentation (https://keras.io/api/losses/)，但是关于计算损失的输出形状，我必须跳过某些内容。

使用此自定义函数训练模型会产生以下错误：

model.compile(loss=spearman_correlation, optimizer=tf.keras.optimizers.Adam())
model.fit(train_x, train_y,batch_size=64, epochs=2, validation_data=(test_x, test_y), callbacks=[model_checkpoint])

InvalidArgumentError:  In[1] is not a matrix. Instead it has shape []
     [[node gradient_tape/model_19/dense_19/MatMul_1 (defined at <ipython-input-46-7e6fc7cd1b39>:12) ]] [Op:__inference_train_function_300522]

我尝试了一种棘手的方法来解决这个问题，我使用了一个 Keras 损失函数的工作示例，我只是用我的损失函数中计算的值修改了结果。这样训练功能就可以工作了，但是，我认为这不是正确的做事方式，但我看不出问题出在哪里。查看自定义函数中的打印输出，可以看出我的损失输出对象的形状和类型与tensorflow的损失函数输出对象相同。

这是我计算损失的方式：

def get_rank(y_pred):
    temp = sorted(y_pred, reverse=False)
    res = [temp.index(i) for i in y_pred]
    res = np.array(res)+1
    return(res)

def custom_spearman_correlation(y_true, y_pred):
    s_coefs = tf.map_fn(lambda k: 1-stats.spearmanr(k[0], get_rank(k[1]))[0], tf.stack([y_true, y_pred], 1), dtype=tf.float32)

    loss = s_coefs
    print("CUSTOM LOSS: ")
    print("Shape: " + str(loss.shape))
    print(type(loss))

    print("WORKING LOSS")
    squared_difference = tf.square(y_true - y_pred)
    w_loss = tf.reduce_mean(squared_difference, axis=-1)
    print("Shape: " + str(w_loss.shape))
    print(type(w_loss))

    print("TRICKY ANSWER: ")
    t_loss = w_loss*0 + loss
    print("Shape: " + str(t_loss.shape))
    print(type(t_loss))
    return loss
    #return w_loss
    #return t_loss

def spearman_correlation(y_true, y_pred):
    sp = tf.py_function(custom_spearman_correlation, [tf.cast(y_true, tf.float32), tf.cast(y_pred, tf.float32)], Tout = tf.float32)
    return (sp)

这是输出：

CUSTOM LOSS: 
Shape: (64,)
<class 'tensorflow.python.framework.ops.EagerTensor'>
WORKING LOSS
Shape: (64,)
<class 'tensorflow.python.framework.ops.EagerTensor'>
TRICKY ANSWER: 
Shape: (64,)

Answer 1

虽然我不确定，但我认为上述解决方案不允许正确更新模型中不同参数的权重，因此我的模型没有学习。我一直在努力按照本网站的定义 (https://rpubs.com/aaronsc32/spearman-rank-correlation) 在 tensorflow 中直接实现 Spearman 等级相关系数，并且我已经达到了以下代码（我分享它以防万一有人发现它有用）。

@tf.function
def get_rank(y_pred):
  rank = tf.argsort(tf.argsort(y_pred, axis=-1, direction="ASCENDING"), axis=-1)+1 #+1 to get the rank starting in 1 instead of 0
  return rank

@tf.function
def sp_rank(x, y):
  cov = tfp.stats.covariance(x, y, sample_axis=0, event_axis=None)
  sd_x = tfp.stats.stddev(x, sample_axis=0, keepdims=False, name=None)
  sd_y = tfp.stats.stddev(y, sample_axis=0, keepdims=False, name=None)
  return 1-cov/(sd_x*sd_y) #1- because we want to minimize loss

@tf.function
def spearman_correlation(y_true, y_pred):
    #First we obtain the ranking of the predicted values
    y_pred_rank = tf.map_fn(lambda x: get_rank(x), y_pred, dtype=tf.float32)
    
    #Spearman rank correlation between each pair of samples:
    #Sample dim: (1, 8)
    #Batch of samples dim: (None, 8) None=batch_size=64
    #Output dim: (batch_size, ) = (64, )
    sp = tf.map_fn(lambda x: sp_rank(x[0],x[1]), (y_true, y_pred_rank), dtype=tf.float32)
    #Reduce to a single value
    loss = tf.reduce_mean(sp)
    return loss

Keras Custom Loss Function InvalidArgumentError: In[1] is not a matrix. Instead it has shape []

Keras Custom Loss Function InvalidArgumentError: In[1] is not a matrix. Instead it has shape []

python

keras

tensorflow

loss-function