如何在可变范围内重用 LSTM 层和变量（注意力机制）

Question

我的代码中有一个问题，我想在 lstm_decoder 中共享权重（因此基本上只使用一个 LSTM）。我知道网上有一些资源，但我仍然无法理解为什么以下内容不共享权重：

initial_input = tf.unstack(tf.zeros(shape=(1,1,hidden_size2)))

for index in range(window_size):
    with tf.variable_scope('lstm_cell_decoder', reuse = index > 0):
        rnn_decoder_cell = tf.nn.rnn_cell.LSTMCell(hidden_size, state_is_tuple = True)

        output_decoder, state_decoder = tf.nn.static_rnn(rnn_decoder_cell, initial_input, initial_state=last_encoder_state, dtype=tf.float32)

        # Compute the score for source output vector
        scores = tf.matmul(concat_lstm_outputs, tf.reshape(output_decoder[-1],(hidden_size,1)))
        attention_coef = tf.nn.softmax(scores)
        context_vector = tf.reduce_sum(tf.multiply(concat_lstm_outputs, tf.reshape(attention_coef, (window_size, 1))),0)
        context_vector = tf.reshape(context_vector, (1,hidden_size))

        # compute the tilda hidden state \tilde{h}_t=tanh(W[c_t, h_t]+b_t)
        concat_context = tf.concat([context_vector, output_decoder[-1]], axis = 1)
        W_tilde = tf.Variable(tf.random_normal(shape = [hidden_size*2, hidden_size2], stddev = 0.1), name = "weights_tilde", trainable = True)
        b_tilde = tf.Variable(tf.zeros([1, hidden_size2]), name="bias_tilde", trainable = True)
        hidden_tilde = tf.nn.tanh(tf.matmul(concat_context, W_tilde)+b_tilde) # hidden_tilde is [1*64]

        # update for next time step
        initial_input = tf.unstack(tf.reshape(hidden_tilde, (1,1,hidden_size2)))
        last_encoder_state = state_decoder
        print(initial_input, last_encoder_state)

        # predict the target
        W_target = tf.Variable(tf.random_normal(shape = [hidden_size2, 1], stddev = 0.1), name = "weights_target", trainable = True)
        print(W_target)
        logit = tf.matmul(hidden_tilde, W_target)
        logits = tf.concat([logits, logit], axis = 0)

logits = logits[1:]

我想对每个循环迭代使用相同的 LSTM 单元和相同的 W_target。但是，我在循环中得到 print(initial_input, last_encoder_state) 和 print(W_target) 的 window_size = 2 的以下输出。

[<tf.Tensor 'lstm_cell_decoder/unstack:0' shape=(1, 64) dtype=float32>] 
LSTMStateTuple(c=<tf.Tensor 
'lstm_cell_decoder/rnn/rnn/lstm_cell/lstm_cell/add_1:0' shape=(1, 64) 
dtype=float32>, h=<tf.Tensor 
'lstm_cell_decoder/rnn/rnn/lstm_cell/lstm_cell/mul_2:0' shape=(1, 64) 
dtype=float32>)
<tf.Variable 'lstm_cell_decoder/weights_target:0' shape=(64, 1) 
dtype=float32_ref>
[<tf.Tensor 'lstm_cell_decoder_1/unstack:0' shape=(1, 64) dtype=float32>] 
LSTMStateTuple(c=<tf.Tensor 
'lstm_cell_decoder_1/rnn/rnn/lstm_cell/lstm_cell/add_1:0' shape=(1, 64) 
dtype=float32>, h=<tf.Tensor 
'lstm_cell_decoder_1/rnn/rnn/lstm_cell/lstm_cell/mul_2:0' shape=(1, 64) 
dtype=float32>)
<tf.Variable 'lstm_cell_decoder_1/weights_target:0' shape=(64, 1) 
dtype=float32_ref>

更新：根据 Maxim 的评论，我尝试了以下语法

for index in range(window_size):
  with tf.variable_scope('lstm_cell_decoder', reuse = index > 0):
     rnn_decoder_cell = tf.nn.rnn_cell.LSTMCell(hidden_size,reuse=index > 0)
     output_decoder, state_decoder = tf.nn.static_rnn(rnn_decoder_cell, ...)
     W_target = tf.get_variable(...)

它现在可以正确地共享变量 W_target 但共享 lstm 仍然存在问题 cell/weights:

<tf.Tensor 'lstm_cell_decoder/rnn/rnn/lstm_cell/lstm_cell/mul_2:0' shape=(1, 
 64) dtype=float32>]
 LSTMStateTuple(c=<tf.Tensor 
 'lstm_cell_decoder/rnn/rnn/lstm_cell/lstm_cell/add_1:0' shape=(1, 64) 
 dtype=float32>, h=<tf.Tensor 
'lstm_cell_decoder/rnn/rnn/lstm_cell/lstm_cell/mul_2:0' shape=(1, 64) 
 dtype=float32>)
 <tf.Variable 'lstm_cell_decoder/weights_target:0' shape=(64, 1) 
 dtype=float32_ref>

 [<tf.Tensor 'lstm_cell_decoder_1/rnn/rnn/lstm_cell/lstm_cell/mul_2:0' 
 shape=(1, 64) dtype=float32>]
 LSTMStateTuple(c=<tf.Tensor 
 'lstm_cell_decoder_1/rnn/rnn/lstm_cell/lstm_cell/add_1:0' shape=(1, 64) 
 dtype=float32>, h=<tf.Tensor 
 'lstm_cell_decoder_1/rnn/rnn/lstm_cell/lstm_cell/mul_2:0' shape=(1, 64) 
 dtype=float32>)
 <tf.Variable 'lstm_cell_decoder/weights_target:0' shape=(64, 1) 
 dtype=float32_ref>

Answer 1

首先，使用 tf.Variable 创建变量不会使其可重用。这是 tf.Variable and tf.get_variable 之间的主要区别之一。看这个例子：

with tf.variable_scope('foo', reuse=tf.AUTO_REUSE):
  for i in range(3):
    x = tf.Variable(0.0, name='x')
    y = tf.get_variable(name='y', shape=())

如果您检查创建的变量，您会看到：

<tf.Variable 'foo/x:0' shape=() dtype=float32_ref>
<tf.Variable 'foo/y:0' shape=() dtype=float32_ref>
<tf.Variable 'foo/x_1:0' shape=() dtype=float32_ref>
<tf.Variable 'foo/x_2:0' shape=() dtype=float32_ref>

接下来，RNN 单元提供自己的重用机制。例如，对于 tf.nn.rnn_cell.LSTMCell 它是 reuse 构造函数参数：

reuse = tf.AUTO_REUSE  # Try also True and False
cell1 = tf.nn.rnn_cell.LSTMCell(3, reuse=reuse)
cell2 = tf.nn.rnn_cell.LSTMCell(3, reuse=reuse)
outputs1, states1 = tf.nn.dynamic_rnn(cell1, X, dtype=tf.float32)
outputs2, states2 = tf.nn.dynamic_rnn(cell2, X, dtype=tf.float32)

如何在可变范围内重用 LSTM 层和变量（注意力机制）

How to reuse LSTM layer and variables in variable scope (attention mechanism)

tensorflow

machine-learning

scope

lstm

attention-model