如何在张量流中累积梯度？

Question

我有一个类似于 this one 的问题。

因为我的资源有限并且我使用深度模型 (VGG-16) - 用于训练三元组网络 - 我想为 128 批大小为 1 的训练示例累积梯度，然后传播错误和更新权重。

我不清楚该怎么做。我使用 tensorflow，但欢迎任何 implementation/pseudocode。

Answer 1

让我们浏览一下您喜欢的答案之一中提出的代码：

## Optimizer definition - nothing different from any classical example
opt = tf.train.AdamOptimizer()

## Retrieve all trainable variables you defined in your graph
tvs = tf.trainable_variables()
## Creation of a list of variables with the same shape as the trainable ones
# initialized with 0s
accum_vars = [tf.Variable(tf.zeros_like(tv.initialized_value()), trainable=False) for tv in tvs]
zero_ops = [tv.assign(tf.zeros_like(tv)) for tv in accum_vars]

## Calls the compute_gradients function of the optimizer to obtain... the list of gradients
gvs = opt.compute_gradients(rmse, tvs)

## Adds to each element from the list you initialized earlier with zeros its gradient (works because accum_vars and gvs are in the same order)
accum_ops = [accum_vars[i].assign_add(gv[0]) for i, gv in enumerate(gvs)]

## Define the training step (part with variable value update)
train_step = opt.apply_gradients([(accum_vars[i], gv[1]) for i, gv in enumerate(gvs)])

第一部分主要是向图表添加新的 variables 和 ops，这样您就可以

在（列表）变量 accum_vars

accum_ops

使用 ops train_step

然后，要在训练时使用它，您必须按照以下步骤操作（仍然来自您链接的答案）：

## The while loop for training
while ...:
    # Run the zero_ops to initialize it
    sess.run(zero_ops)
    # Accumulate the gradients 'n_minibatches' times in accum_vars using accum_ops
    for i in xrange(n_minibatches):
        sess.run(accum_ops, feed_dict=dict(X: Xs[i], y: ys[i]))
    # Run the train_step ops to update the weights based on your accumulated gradients
    sess.run(train_step)

Answer 2

Tensorflow 2.0 Compatible Answer: 与上述Pop's Answer和Tensorflow Website中的解释相符，下面是Tensorflow Version中Accumulating Gradients的代码2.0:

def train(epochs):
  for epoch in range(epochs):
    for (batch, (images, labels)) in enumerate(dataset):
       with tf.GradientTape() as tape:
        logits = mnist_model(images, training=True)
        tvs = mnist_model.trainable_variables
        accum_vars = [tf.Variable(tf.zeros_like(tv.initialized_value()), trainable=False) for tv in tvs]
        zero_ops = [tv.assign(tf.zeros_like(tv)) for tv in accum_vars]
        loss_value = loss_object(labels, logits)

       loss_history.append(loss_value.numpy().mean())
       grads = tape.gradient(loss_value, tvs)
       #print(grads[0].shape)
       #print(accum_vars[0].shape)
       accum_ops = [accum_vars[i].assign_add(grad) for i, grad in enumerate(grads)]



    optimizer.apply_gradients(zip(grads, mnist_model.trainable_variables))
    print ('Epoch {} finished'.format(epoch))

# call the above function    
train(epochs = 3)

完整代码见Github Gist.

如何在张量流中累积梯度？

How to accumulate gradients in tensorflow?

gradient-descent

conv-neural-network

tensorflow