如何对词向量进行聚类和分类

Question

我目前正在训练一个 skip-gram 模型，通过描述来学习不同的对象。在我从那个模型中得到我的词嵌入之后，我想将它们聚集在相似的组中并标记它们。

我的想法是重复使用具有相同嵌入层的相同模型，并让它从它们的描述中学习类别。

这是我目前的结果：

问题是新类别被标记为 60、61 和 62。该模型将它们解释为相似并将它们放在相同的 space.

这些类别不应该相同，并且它们不接近应有的向量。我做错了吗？我如何重用我的模型来对这些对象进行聚类和分类？

pretrained_vectors_cat =
array([[-0.00703605, -0.00456019, -0.07583138, ..., -0.00803135,
        -0.03794867, -0.03410311],
       [-0.06226502, -0.03059928, -0.07528683, ...,  0.11714505,
         0.01752528, -0.00584977],
       [-0.07654897, -0.04235281, -0.02850686, ...,  0.06900358,
         0.00327334, -0.10425693],
       ...,
       [-0.50258852, -0.57102433, -0.28687169, ..., -0.26322143,
        -0.0910767 ,  0.13004072],
       [-0.53029969,  0.71982554, -0.80099767, ...,  0.75670917,
        -0.61081131,  0.59293241],
       [ 0.22630654, -0.69713363, -0.1661163 , ..., -0.23165715,
         0.18017072, -0.90354915]])

with graph_pretrained.as_default():

    with tf.name_scope('inputs'):
        train_inputs = tf.placeholder(tf.int32, shape=[batch_size], name="train_inputs")
        train_labels = tf.placeholder(tf.int32, shape=[batch_size, 1], name="train_labels")

    with tf.device(device_name):
        with tf.name_scope('embeddings_pretrained'):
            embeddings = tf.get_variable("embeddings", initializer=pretrained_vectors_cat)
            embed = tf.nn.embedding_lookup(embeddings, train_inputs)

            embeddings = tf.cast(embeddings, tf.float32)
            embed = tf.cast(embed, tf.float32)

        with tf.name_scope('weights'):
            nce_weights = tf.Variable(tf.truncated_normal(shape=[vocabulary_size_cat, embedding_size],
                                                          stddev=1.0 / math.sqrt(embedding_size)), 
                                      name="weight_matrix")

        with tf.name_scope('biases'):
            nce_biases = tf.Variable(tf.zeros([vocabulary_size_cat]), name="bias_matrix")

    with tf.name_scope('loss'): 
        loss = tf.reduce_mean(tf.nn.nce_loss(
                weights=nce_weights,
                biases=nce_biases,
                inputs=embed,
                labels=train_labels,
                num_sampled=num_sampled,
                num_classes=vocabulary_size_cat))

    loss_summary = tf.summary.scalar('loss', loss)

    with tf.name_scope('optimizer'):
        optimizer = tf.train.GradientDescentOptimizer(learningrate).minimize(loss)

    norm = tf.sqrt(tf.reduce_sum(tf.square(embeddings), 1, keepdims=True))
    normalized_embeddings = embeddings / norm

    merged = tf.summary.merge_all()

    init = tf.global_variables_initializer()

    saver = tf.train.Saver()

with tf.Session(graph=graph_pretrained, config=session_config) as session:
    # Open a writer to write summaries.
    writer = tf.summary.FileWriter(log_dir + "/", session.graph)
    writer_loss = tf.summary.FileWriter(log_dir + "/loss {}".format(model_name))
    init.run()
    average_loss = 0

    for step in xrange(num_steps):      
        progbar.update(step)

        batch_inputs, batch_labels = generateCenterContextBatch(batch_size, window_size, new_project_mark)

        feed_dict = {train_inputs: batch_inputs, train_labels: batch_labels}#, embedding_placeholder: pretrained_vectors}

        run_metadata = tf.RunMetadata()
        _, summary, loss_val = session.run([optimizer, merged, loss],
                                            feed_dict=feed_dict,
                                            run_metadata=run_metadata)

        average_loss += loss_val

        # Add returned summaries to writer in each step.
        writer_loss.add_summary(summary, step)

        # Add metadata to visualize the graph for the last run.
        if step == (num_steps - 1):
            writer_loss.add_run_metadata(run_metadata, 'step%d' % step)

    final_embeddings_category = normalized_embeddings.eval()

    # Save the model for checkpoints.
    saver.save(session, os.path.join(logdir_model", 'model.ckpt'))

writer_loss.close()

Answer 1

问题已解决。

TensorBoard Projector 计算的余弦距离有误。通过 sklearn 获取距离会产生正确的聚类。

TensorBoard 可能会将原始维度从 300 减少到 200，并计算与减少的维度的距离。所以标签 "nearest points in the original space" 具有误导性。

--- 检查：installed_packages
信息：已安装：tensorboard==1.13.1
信息：已安装：tensorflow-gpu==1.13.1
信息：已安装：tensorflow==1.14.0

如何对词向量进行聚类和分类

How to cluster and classify word-vectors

python

cluster-analysis

word2vec