gensim doc2vec 嵌入向量的差异

Question

我使用 gensim Doc2Vec 包来训练 doc2vec 嵌入。我希望使用相同参数和数据训练的两个模型具有非常接近的 doc2vec 向量值。然而，根据我的经验，只有在没有训练词嵌入（dbow_words = 0）的情况下，在 PV-DBOW 中训练的 doc2vec 才是正确的。对于 dbow_words = 1 的 PV-DM 和 PV-DBOW，即在每种情况下词嵌入都与 doc2vec 一起训练，相同训练模型的 doc2vec 嵌入向量是相当不同的。

这是我的代码

    from sklearn.datasets import fetch_20newsgroups
    from gensim import models
    import scipy.spatial.distance as distance
    import numpy as np
    from nltk.corpus import stopwords
    from string import punctuation
    def clean_text(texts,  min_length = 2):
        clean = []
        #don't remove apostrophes
        translator = str.maketrans(punctuation.replace('\'',' '), ' '*len(punctuation))
        for text in texts:
            text = text.translate(translator)
            tokens = text.split()
            # remove not alphabetic tokens
            tokens = [word.lower() for word in tokens if word.isalpha()]
            # filter out stop words
            stop_words = stopwords.words('english')
            tokens = [w for w in tokens if not w in stop_words]
            # filter out short tokens
            tokens = [word for word in tokens if len(word) >= min_length]
            tokens = ' '.join(tokens)
            clean.append(tokens)
        return clean
    def tag_text(all_text, tag_type =''):
        tagged_text = []
        for i, text in enumerate(all_text):
            tag = tag_type + '_' + str(i)
            tagged_text.append(models.doc2vec.TaggedDocument(text.split(), [tag]))
        return tagged_text

    def train_docvec(dm, dbow_words, min_count, epochs, training_data):
        model = models.Doc2Vec(dm=dm, dbow_words = dbow_words, min_count = min_count)
        model.build_vocab(tagged_data)
        model.train(training_data, total_examples=len(training_data), epochs=epochs)    
        return model

    def compare_vectors(vector1, vector2):
        cos_distances = []
        for i in range(len(vector1)):
            d = distance.cosine(vector1[i], vector2[i])
            cos_distances.append(d)
        print (np.median(cos_distances))
        print (np.std(cos_distances))    

    dataset = fetch_20newsgroups(shuffle=True, random_state=1,remove=('headers', 'footers', 'quotes'))
    n_samples = len(dataset.data)
    data = clean_text(dataset.data)
    tagged_data = tag_text(data)
    data_labels = dataset.target
    data_label_names = dataset.target_names

    model_dbow1 = train_docvec(0, 0, 4, 30, tagged_data)
    model_dbow2 = train_docvec(0, 0, 4, 30, tagged_data)
    model_dbow3 = train_docvec(0, 1, 4, 30, tagged_data)
    model_dbow4 = train_docvec(0, 1, 4, 30, tagged_data)
    model_dm1 = train_docvec(1, 0, 4, 30, tagged_data)
    model_dm2 = train_docvec(1, 0, 4, 30, tagged_data)

    compare_vectors(model_dbow1.docvecs, model_dbow2.docvecs)
    > 0.07795828580856323
    > 0.02610614028793008

    compare_vectors(model_dbow1.docvecs, model_dbow3.docvecs)
    > 0.6476179957389832
    > 0.14797587172616306

    compare_vectors(model_dbow3.docvecs, model_dbow4.docvecs)
    > 0.19878000020980835
    > 0.06362519480831186

    compare_vectors(model_dm1.docvecs, model_dm2.docvecs)
    > 0.13536489009857178
    > 0.045365127475424386

    compare_vectors(model_dbow1.docvecs, model_dm1.docvecs)
    > 0.6358324736356735
    > 0.15150255674571805

UPDATE

按照 gojomo 的建议，我尝试比较向量之间的差异，不幸的是，这些差异更糟：

def compare_vector_differences(vector1, vector2):
    diff1 = []
    diff2 = []
    for i in range(len(vector1)-1):
        diff1.append( vector1[i+1] - vector1[i])
    for i in range(len(vector2)-1):
        diff2[i].append(vector2[i+1] - vector2[i])
    cos_distances = []
    for i in range(len(diff1)):
        d = distance.cosine(diff1[i], diff2[i])
        cos_distances.append(d)
    print (np.median(cos_distances))
    print (np.std(cos_distances))    

compare_vector_differences(model_dbow1.docvecs, model_dbow2.docvecs)
> 0.1134452223777771
> 0.02676398444178949

compare_vector_differences(model_dbow1.docvecs, model_dbow3.docvecs)
> 0.8464127033948898
> 0.11423789350773429

compare_vector_differences(model_dbow4.docvecs, model_dbow3.docvecs)

> 0.27400463819503784
> 0.05984108730423529

SECOND UPDATE

这次终于看懂了gojomo，东西看起来就好了

def compare_distance_differences(vector1, vector2):
    diff1 = []
    diff2 = []
    for i in range(len(vector1)-1):
        diff1.append( distance.cosine(vector1[i+1], vector1[i]))
    for i in range(len(vector2)-1):
        diff2.append( distance.cosine(vector2[i+1], vector2[i]))
    diff_distances = []
    for i in range(len(diff1)):
        diff_distances.append(abs(diff1[i] - diff2[i]))
    print (np.median(diff_distances))
    print (np.std(diff_distances))    

compare_distance_differences(model_dbow1.docvecs, model_dbow2.docvecs)
>0.017469733953475952
>0.01659284710785352

compare_distance_differences(model_dbow1.docvecs, model_dbow3.docvecs)
>0.0786697268486023
>0.06092163158218411

compare_distance_differences(model_dbow3.docvecs, model_dbow4.docvecs)
>0.02321992814540863
>0.023095123172320778

Answer 1

Doc2Vec 和 Word2Vec 模型的 doc-vectors（或 word-vectors）仅与 co-trained交错的培训课程。

否则，算法 (random-initialization & random-sampling) 引入的随机性和训练顺序的细微差异（来自多线程）将导致单个向量的训练位置漂移到任意不同的位置.他们的 relative distances/directions，相对于共享交错训练的其他向量，从一个模型到下一个模型应该大约 equally-useful。

但是这样的向量没有一个正确的位置，并且测量一个模型中文档“1”（或单词 'foo'）的向量与另一个模型中的相应向量之间的差异，是'反映 models/algorithms 受过训练提供的任何内容。

Gensim 常见问题解答中有更多信息：

Q11: I've trained my Word2Vec/Doc2Vec/etc model repeatedly using the exact same text corpus, but the vectors are different each time. Is there a bug or have I made a mistake?

gensim doc2vec 嵌入向量的差异

Discrepancies in gensim doc2vec embedding vectors

gensim

word-embedding

doc2vec