TensorFlow：训练循环中的每次迭代都比较慢

Question

我正在 TensorFlow 中训练具有三个隐藏层的标准、简单的多层感知器 ANN。我添加了一个文本进度条，这样我就可以看到遍历各个时期的进度。我发现每次迭代的处理时间在前几个时期后增加。这是显示每次迭代增加的示例屏幕截图：

在这种情况下，前几次迭代大约花费了 1.05 秒/它，到 100% 时它花费了 4.01 秒/它。

此处列出相关代码：

# ------------------------- Build the TensorFlow Graph -------------------------

with tf.Graph().as_default():

    (a bunch of statements for specifying the graph)

# --------------------------------- Training ----------------------------------

    sess = tf.InteractiveSession()
    sess.run(tf.initialize_all_variables())

    print "Start Training"

    pbar = tqdm(total = training_epochs)
    for epoch in range(training_epochs):
        avg_cost = 0.0
    batch_iter = 0

    while batch_iter < batch_size:
        train_features = []
        train_labels = []
        batch_segments = random.sample(train_segments, 20)
        for segment in batch_segments:
            train_features.append(segment[0])
            train_labels.append(segment[1])
        sess.run(optimizer, feed_dict={x: train_features, y_: train_labels})
        line_out = "," + str(batch_iter) + "\n"
        train_outfile.write(line_out)
        line_out = ",," + str(sess.run(tf.reduce_mean(weights['h1']), feed_dict={x: train_features, y_: train_labels}))
        line_out += "," + str(sess.run(tf.reduce_mean(weights['h2']), feed_dict={x: train_features, y_: train_labels}))
        line_out += "," + str(sess.run(tf.reduce_mean(weights['h3']), feed_dict={x: train_features, y_: train_labels})) + "\n"
        train_outfile.write(line_out)
        avg_cost += sess.run(cost, feed_dict={x: train_features, y_: train_labels})/batch_size

        batch_iter += 1

    pbar.update(1)  # Increment the progress bar by one

train_outfile.close()
print "Completed training"

在搜索 Whosebug 时，我发现其他人也遇到了每次迭代花费的时间比上一次长的问题。但是，我相信我的可能有所不同，因为他们使用如下语句明确地将操作添加到图中：

distorted_image = tf.image.random_flip_left_right(image_tensor)

虽然我是 TensorFlow 的新手，但我认为我不会犯同样的错误，因为我的循环中唯一的东西是 sess.run() 调用。

非常感谢任何帮助。

Answer 1

你拥有的三个地方：

sess.run(tf.reduce_mean(weights['h1']), ...)

在 while 循环的每次迭代中，每个 tf.reduce_mean() 节点都将一个新的 tf.reduce_mean() 节点附加到图 ，这会增加开销。尝试在 while 循环之外创建它们：

with tf.Graph().as_default(): ... m1 = tf.reduce_mean(weights['h1']) while batch_iter < batch_size: ... line_out = ",," + str(sess.run(m1, feed_dict={x: train_features, y_: train_labels}))

TensorFlow：训练循环中的每次迭代都比较慢

TensorFlow: Each iteration in training for-loop slower

python

for-loop

tensorflow