tf.compat.v1.train.exponential_decay: 全局步长 = 0
tf.compat.v1.train.exponential_decay: global step = 0
为了了解如何实现具有指数衰减和恒定学习率的 ANN，我在此处进行了查找：https://www.tensorflow.org/api_docs/python/tf/compat/v1/train/exponential_decay
我有一些问题：
...
global_step = tf.Variable(0, trainable=False)
starter_learning_rate = 0.1
learning_rate = tf.compat.v1.train.exponential_decay(starter_learning_rate,
global_step,
100000, 0.96, staircase=True)
# Passing global_step to minimize() will increment it at each step.
learning_step = (
tf.compat.v1.train.GradientDescentOptimizer(learning_rate)
.minimize(...my loss..., global_step=global_step)
)
当 global_step 设置为等于值为 0 的变量时，并不意味着我们不会衰减，因为
decayed_learning_rate = learning_rate *
decay_rate ^ (global_step / decay_steps)
因此如果 global_step= 0 跟在 decayed_learning_rate = learning_rate 之后，这是正确的还是我在这里犯了错误？
此外，我对100,000步到底指的是什么感到有点困惑。一步到底是什么？是每次输入完全通过网络并反向传播时吗？
我希望这个例子能消除你的疑虑。
epochs = 10
global_step = tf.Variable(0, trainable=False, dtype= tf.int32)
starter_learning_rate = 1.0
for epoch in range(epochs):
print("Starting Epoch {}/{}".format(epoch+1,epochs))
for step, (x_batch_train, y_batch_train) in enumerate(train_dataset):
with tf.GradientTape() as tape:
logits = model(x_batch_train, training=True)
loss_value = loss_fn(y_batch_train, logits)
grads = tape.gradient(loss_value, model.trainable_weights)
learning_rate = tf.compat.v1.train.exponential_decay(
starter_learning_rate,
global_step,
100000,
0.96
)
optimizer(learning_rate=learning_rate).apply_gradients(zip(grads, model.trainable_weights))
print("Global Step: {} Learning Rate: {} Examples Processed: {}".format(global_step.numpy(), learning_rate(), (step + 1) * 100))
global_step.assign_add(1)
输出：
Starting Epoch 1/10
Global Step: 0 Learning Rate: 1.0 Examples Processed: 100
Global Step: 1 Learning Rate: 0.9999996423721313 Examples Processed: 200
Global Step: 2 Learning Rate: 0.9999992251396179 Examples Processed: 300
Global Step: 3 Learning Rate: 0.9999988079071045 Examples Processed: 400
Global Step: 4 Learning Rate: 0.9999983906745911 Examples Processed: 500
Global Step: 5 Learning Rate: 0.9999979734420776 Examples Processed: 600
Global Step: 6 Learning Rate: 0.9999975562095642 Examples Processed: 700
Global Step: 7 Learning Rate: 0.9999971389770508 Examples Processed: 800
Global Step: 8 Learning Rate: 0.9999967217445374 Examples Processed: 900
Global Step: 9 Learning Rate: 0.9999963045120239 Examples Processed: 1000
Global Step: 10 Learning Rate: 0.9999958872795105 Examples Processed: 1100
Global Step: 11 Learning Rate: 0.9999954700469971 Examples Processed: 1200
Starting Epoch 2/10
Global Step: 12 Learning Rate: 0.9999950528144836 Examples Processed: 100
Global Step: 13 Learning Rate: 0.9999946355819702 Examples Processed: 200
Global Step: 14 Learning Rate: 0.9999942183494568 Examples Processed: 300
Global Step: 15 Learning Rate: 0.9999938607215881 Examples Processed: 400
Global Step: 16 Learning Rate: 0.9999934434890747 Examples Processed: 500
Global Step: 17 Learning Rate: 0.999993085861206 Examples Processed: 600
Global Step: 18 Learning Rate: 0.9999926686286926 Examples Processed: 700
Global Step: 19 Learning Rate: 0.9999922513961792 Examples Processed: 800
Global Step: 20 Learning Rate: 0.9999918341636658 Examples Processed: 900
Global Step: 21 Learning Rate: 0.9999914169311523 Examples Processed: 1000
Global Step: 22 Learning Rate: 0.9999909996986389 Examples Processed: 1100
Global Step: 23 Learning Rate: 0.9999905824661255 Examples Processed: 1200
现在，如果您将全局步骤保持为 0。即从上面的代码中删除增量操作。
输出：
开始纪元 1/10
Global Step: 0 Learning Rate: 1.0 Examples Processed: 100
Global Step: 0 Learning Rate: 1.0 Examples Processed: 200
Global Step: 0 Learning Rate: 1.0 Examples Processed: 300
Global Step: 0 Learning Rate: 1.0 Examples Processed: 400
Global Step: 0 Learning Rate: 1.0 Examples Processed: 500
Global Step: 0 Learning Rate: 1.0 Examples Processed: 600
Global Step: 0 Learning Rate: 1.0 Examples Processed: 700
Global Step: 0 Learning Rate: 1.0 Examples Processed: 800
Global Step: 0 Learning Rate: 1.0 Examples Processed: 900
Global Step: 0 Learning Rate: 1.0 Examples Processed: 1000
Global Step: 0 Learning Rate: 1.0 Examples Processed: 1100
Global Step: 0 Learning Rate: 1.0 Examples Processed: 1200
Starting Epoch 2/10
Global Step: 0 Learning Rate: 1.0 Examples Processed: 100
Global Step: 0 Learning Rate: 1.0 Examples Processed: 200
Global Step: 0 Learning Rate: 1.0 Examples Processed: 300
Global Step: 0 Learning Rate: 1.0 Examples Processed: 400
Global Step: 0 Learning Rate: 1.0 Examples Processed: 500
Global Step: 0 Learning Rate: 1.0 Examples Processed: 600
Global Step: 0 Learning Rate: 1.0 Examples Processed: 700
Global Step: 0 Learning Rate: 1.0 Examples Processed: 800
Global Step: 0 Learning Rate: 1.0 Examples Processed: 900
Global Step: 0 Learning Rate: 1.0 Examples Processed: 1000
Global Step: 0 Learning Rate: 1.0 Examples Processed: 1100
Global Step: 0 Learning Rate: 1.0 Examples Processed: 1200
建议 - 不要使用 tf.compat。v1.train.exponential_decay 使用 tf.keras.optimizers.schedules.ExponentialDecay。
这就是最简单的例子。
def create_model1():
initial_learning_rate = 0.01
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
initial_learning_rate,
decay_steps=100000,
decay_rate=0.96,
staircase=True)
model = tf.keras.Sequential()
model.add(tf.keras.Input(shape=(5,)))
model.add(tf.keras.layers.Dense(units = 6,
activation='relu',
name = 'd1'))
model.add(tf.keras.layers.Dense(units = 2, activation='softmax', name = 'O2'))
model.compile(optimizer=tf.keras.optimizers.SGD(learning_rate=lr_schedule),
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
return model
model = create_model1()
model.fit(x, y, batch_size = 100, epochs = 100)
你也可以使用像tf.keras.callbacks.LearningRateScheduler这样的回调来实现你的衰减。
为了了解如何实现具有指数衰减和恒定学习率的 ANN，我在此处进行了查找：https://www.tensorflow.org/api_docs/python/tf/compat/v1/train/exponential_decay
我有一些问题：
...
global_step = tf.Variable(0, trainable=False)
starter_learning_rate = 0.1
learning_rate = tf.compat.v1.train.exponential_decay(starter_learning_rate,
global_step,
100000, 0.96, staircase=True)
# Passing global_step to minimize() will increment it at each step.
learning_step = (
tf.compat.v1.train.GradientDescentOptimizer(learning_rate)
.minimize(...my loss..., global_step=global_step)
)
当 global_step 设置为等于值为 0 的变量时，并不意味着我们不会衰减，因为
decayed_learning_rate = learning_rate *
decay_rate ^ (global_step / decay_steps)
因此如果 global_step= 0 跟在 decayed_learning_rate = learning_rate 之后，这是正确的还是我在这里犯了错误？
此外，我对100,000步到底指的是什么感到有点困惑。一步到底是什么？是每次输入完全通过网络并反向传播时吗？
我希望这个例子能消除你的疑虑。
epochs = 10
global_step = tf.Variable(0, trainable=False, dtype= tf.int32)
starter_learning_rate = 1.0
for epoch in range(epochs):
print("Starting Epoch {}/{}".format(epoch+1,epochs))
for step, (x_batch_train, y_batch_train) in enumerate(train_dataset):
with tf.GradientTape() as tape:
logits = model(x_batch_train, training=True)
loss_value = loss_fn(y_batch_train, logits)
grads = tape.gradient(loss_value, model.trainable_weights)
learning_rate = tf.compat.v1.train.exponential_decay(
starter_learning_rate,
global_step,
100000,
0.96
)
optimizer(learning_rate=learning_rate).apply_gradients(zip(grads, model.trainable_weights))
print("Global Step: {} Learning Rate: {} Examples Processed: {}".format(global_step.numpy(), learning_rate(), (step + 1) * 100))
global_step.assign_add(1)
输出：
Starting Epoch 1/10
Global Step: 0 Learning Rate: 1.0 Examples Processed: 100
Global Step: 1 Learning Rate: 0.9999996423721313 Examples Processed: 200
Global Step: 2 Learning Rate: 0.9999992251396179 Examples Processed: 300
Global Step: 3 Learning Rate: 0.9999988079071045 Examples Processed: 400
Global Step: 4 Learning Rate: 0.9999983906745911 Examples Processed: 500
Global Step: 5 Learning Rate: 0.9999979734420776 Examples Processed: 600
Global Step: 6 Learning Rate: 0.9999975562095642 Examples Processed: 700
Global Step: 7 Learning Rate: 0.9999971389770508 Examples Processed: 800
Global Step: 8 Learning Rate: 0.9999967217445374 Examples Processed: 900
Global Step: 9 Learning Rate: 0.9999963045120239 Examples Processed: 1000
Global Step: 10 Learning Rate: 0.9999958872795105 Examples Processed: 1100
Global Step: 11 Learning Rate: 0.9999954700469971 Examples Processed: 1200
Starting Epoch 2/10
Global Step: 12 Learning Rate: 0.9999950528144836 Examples Processed: 100
Global Step: 13 Learning Rate: 0.9999946355819702 Examples Processed: 200
Global Step: 14 Learning Rate: 0.9999942183494568 Examples Processed: 300
Global Step: 15 Learning Rate: 0.9999938607215881 Examples Processed: 400
Global Step: 16 Learning Rate: 0.9999934434890747 Examples Processed: 500
Global Step: 17 Learning Rate: 0.999993085861206 Examples Processed: 600
Global Step: 18 Learning Rate: 0.9999926686286926 Examples Processed: 700
Global Step: 19 Learning Rate: 0.9999922513961792 Examples Processed: 800
Global Step: 20 Learning Rate: 0.9999918341636658 Examples Processed: 900
Global Step: 21 Learning Rate: 0.9999914169311523 Examples Processed: 1000
Global Step: 22 Learning Rate: 0.9999909996986389 Examples Processed: 1100
Global Step: 23 Learning Rate: 0.9999905824661255 Examples Processed: 1200
现在，如果您将全局步骤保持为 0。即从上面的代码中删除增量操作。 输出：
开始纪元 1/10
Global Step: 0 Learning Rate: 1.0 Examples Processed: 100
Global Step: 0 Learning Rate: 1.0 Examples Processed: 200
Global Step: 0 Learning Rate: 1.0 Examples Processed: 300
Global Step: 0 Learning Rate: 1.0 Examples Processed: 400
Global Step: 0 Learning Rate: 1.0 Examples Processed: 500
Global Step: 0 Learning Rate: 1.0 Examples Processed: 600
Global Step: 0 Learning Rate: 1.0 Examples Processed: 700
Global Step: 0 Learning Rate: 1.0 Examples Processed: 800
Global Step: 0 Learning Rate: 1.0 Examples Processed: 900
Global Step: 0 Learning Rate: 1.0 Examples Processed: 1000
Global Step: 0 Learning Rate: 1.0 Examples Processed: 1100
Global Step: 0 Learning Rate: 1.0 Examples Processed: 1200
Starting Epoch 2/10
Global Step: 0 Learning Rate: 1.0 Examples Processed: 100
Global Step: 0 Learning Rate: 1.0 Examples Processed: 200
Global Step: 0 Learning Rate: 1.0 Examples Processed: 300
Global Step: 0 Learning Rate: 1.0 Examples Processed: 400
Global Step: 0 Learning Rate: 1.0 Examples Processed: 500
Global Step: 0 Learning Rate: 1.0 Examples Processed: 600
Global Step: 0 Learning Rate: 1.0 Examples Processed: 700
Global Step: 0 Learning Rate: 1.0 Examples Processed: 800
Global Step: 0 Learning Rate: 1.0 Examples Processed: 900
Global Step: 0 Learning Rate: 1.0 Examples Processed: 1000
Global Step: 0 Learning Rate: 1.0 Examples Processed: 1100
Global Step: 0 Learning Rate: 1.0 Examples Processed: 1200
建议 - 不要使用 tf.compat。v1.train.exponential_decay 使用 tf.keras.optimizers.schedules.ExponentialDecay。 这就是最简单的例子。
def create_model1():
initial_learning_rate = 0.01
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
initial_learning_rate,
decay_steps=100000,
decay_rate=0.96,
staircase=True)
model = tf.keras.Sequential()
model.add(tf.keras.Input(shape=(5,)))
model.add(tf.keras.layers.Dense(units = 6,
activation='relu',
name = 'd1'))
model.add(tf.keras.layers.Dense(units = 2, activation='softmax', name = 'O2'))
model.compile(optimizer=tf.keras.optimizers.SGD(learning_rate=lr_schedule),
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
return model
model = create_model1()
model.fit(x, y, batch_size = 100, epochs = 100)
你也可以使用像tf.keras.callbacks.LearningRateScheduler这样的回调来实现你的衰减。