如何在 Tensorflow 中使用词嵌入进行预测

Question

我正在努力完成 Tensorflow 教程，但在尝试增强 RNN/language model tutorial 时遇到困难，以便我可以预测句子中的下一个词。本教程使用词嵌入作为词的表示。

由于模型学习词嵌入，我假设我添加的任何类型的预测都会输出相同的嵌入。我想不通的是如何将这些嵌入转换回数据集中的单词 id。我看到的唯一示例保留了一个内存数据结构，其映射与 wordid -> 嵌入相反，并将其用于查找。这显然不适用于所有问题。有没有更好的方法？

Answer 1

假设您同时拥有词汇表中的 word_to_idx 和 idx_to_word，这是您所做操作的伪代码

假设预测的输入是"this is sample"

batch_size = 1
num_steps = 3 # i.e each step for the word in "this is sample"
hidden_size = 1500
vocab_size = 10000
translate the `word_to_idx` for the input `"this is sample"`
get the word embeddings for the each word in the input
Input to model will be word embedding of size 1x1500 at each time step
Output of model at each time step will be of size 1x1500
y_pred is output at last step from model for the given input 
adding projection to output (i.e y_pred x Weights(hidden_size, vocab_size) + bias(vocab_size, )  = 1x10000)
now sample the output through the below function to get the index with max probability
generate idx_to_word from the index we just got
use the generated word along with the previous input to generate next word until you get `<eos>` or some predefined sentence stopping.

这是从 here:

中抽样的示例

def sample(a, temperature=1.0):
    # helper function to sample an index from a probability array
    a = np.log(a) / temperature
    a = np.exp(a) / np.sum(np.exp(a))
    return np.argmax(np.random.multinomial(1, a, 1))

如何在 Tensorflow 中使用词嵌入进行预测

How to use word embeddings for prediction in Tensorflow

tensorflow

word-embedding