Keras/Tensorflow 优化器中的多 GPU InvalidArgumentError

Question

我想尝试使用 Tensorflow 后端在 Keras 中进行多 GPU 训练。

我正在尝试此处描述的函数 make_parallel：https://medium.com/@kuza55/transparent-multi-gpu-training-on-tensorflow-with-keras-8b0016fd9012。代码在这里（针对 Keras 2 更新）：

from keras.layers import concatenate
from keras.layers.core import Lambda
from keras.models import Model

import tensorflow as tf

def make_parallel(model, gpu_count):
    def get_slice(data, idx, parts):
        shape = tf.shape(data)
        size = tf.concat([ shape[:1] // parts, shape[1:] ],axis=0)
        stride = tf.concat([ shape[:1] // parts, shape[1:]*0 ],axis=0)
        start = stride * idx
        return tf.slice(data, start, size)

    outputs_all = []
    for i in range(len(model.outputs)):
        outputs_all.append([])

    #Place a copy of the model on each GPU, each getting a slice of the batch
    for i in range(gpu_count):
        with tf.device('/gpu:%d' % i):
            with tf.name_scope('tower_%d' % i) as scope:

                inputs = []
                #Slice each input into a piece for processing on this GPU
                for x in model.inputs:
                    input_shape = tuple(x.get_shape().as_list())[1:]
                    slice_n = Lambda(get_slice, output_shape=input_shape, arguments={'idx':i,'parts':gpu_count})(x)
                    inputs.append(slice_n)                

                outputs = model(inputs)

                if not isinstance(outputs, list):
                    outputs = [outputs]

                #Save all the outputs for merging back together later
                for l in range(len(outputs)):
                    outputs_all[l].append(outputs[l])

    # merge outputs on CPU
    with tf.device('/cpu:0'):
        merged = []
        for outputs in outputs_all:
            merged.append(concatenate(outputs, axis=0))

        return Model(inputs=model.inputs, outputs=merged)

我创建了一个模型：

model = make_parallel(create_model(...), 4)
model.compile(optimizer='adam', loss='mse', metrics=['mae', 'mse',])

在运行适合它训练一个时期然后崩溃，出现以下异常：

InvalidArgumentError (see above for traceback): Incompatible shapes: [120,1] vs. [122,1]
     [[Node: training_6/Adam/gradients/loss_10/concatenate_7_loss/sub_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _class=["loc:@loss_10/concatenate_7_loss/sub"], _device="/job:localhost/replica:0/task:0/gpu:0"](training_6/Adam/gradients/loss_10/concatenate_7_loss/sub_grad/Shape/_10935, training_6/Adam/gradients/loss_10/concatenate_7_loss/sub_grad/Shape_1)]]
     [[Node: training_6/Adam/gradients/concatenate_7/concat_grad/Slice_1/_11003 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_4728_training_6/Adam/gradients/concatenate_7/concat_grad/Slice_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:1"]()]]

在不同 GPU 上组合模型梯度的阶段出现了问题。异常中不兼容的形状大小在某种程度上与批量大小（此处为 128）相关（即更改批量大小会更改不兼容的形状大小）。

Answer 1

您的问题似乎与报告的问题相似 here。看来输入数据大小必须是GPU数量的倍数。

来自link：

The number of samples just needs to be a mutiple of the total number of GPUs.

Ex. I had 68531 samples in in my input, and once I shaved that down to 68528 with 8 GPUs, it worked fine.

Answer 2

截至 2020 年 12 月，重新排列“MaxPooling2D”layer/s 解决了问题。

Keras/Tensorflow 优化器中的多 GPU InvalidArgumentError

Keras/Tensorflow multi GPU InvalidArgumentError in optimizer

multi-gpu

keras

tensorflow