Word2vec 向量的长度有什么意义？
What meaning does the length of a Word2vec vector have?
我通过 gensim 使用 Word2vec 和 Google 在 Google 新闻上训练的预训练向量。我注意到我可以通过对 Word2Vec 对象进行直接索引查找来访问的词向量不是单位向量：
>>> import numpy
>>> from gensim.models import Word2Vec
>>> w2v = Word2Vec.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
>>> king_vector = w2v['king']
>>> numpy.linalg.norm(king_vector)
2.9022589
但是，在most_similar方法中，没有使用这些非单位向量；相反，使用来自未记录的 .syn0norm 属性 的规范化版本，它仅包含单位向量：
>>> w2v.init_sims()
>>> unit_king_vector = w2v.syn0norm[w2v.vocab['king'].index]
>>> numpy.linalg.norm(unit_king_vector)
0.99999994
更大的向量只是单位向量的放大版本：
>>> king_vector - numpy.linalg.norm(king_vector) * unit_king_vector
array([ 0.00000000e+00, -1.86264515e-09, 0.00000000e+00,
0.00000000e+00, -1.86264515e-09, 0.00000000e+00,
-7.45058060e-09, 0.00000000e+00, 3.72529030e-09,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
... (some lines omitted) ...
-1.86264515e-09, -3.72529030e-09, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00], dtype=float32)
鉴于 Word2Vec 中的单词相似性比较是由 cosine similarity 完成的，对我来说非标准化向量的长度意味着什么并不明显 - 虽然我假设它们意味着 某事 ，因为 gensim 将它们暴露给我，而不是仅在 .syn0norm.
中暴露单位向量
这些非规范化的Word2vec向量的长度是如何生成的，它们的含义是什么？对于哪些计算使用归一化向量有意义，什么时候应该使用非归一化向量？
我认为您正在寻找的答案在 2015 年的论文 Measuring Word Significance 中有所描述
使用
Adriaan Schakel 和 Benjamin Wilson 的分布式单词表示。关键点：
When a word appears
in different contexts, its vector gets moved in
different directions during updates. The final vector
then represents some sort of weighted average
over the various contexts. Averaging over vectors
that point in different directions typically results in
a vector that gets shorter with increasing number
of different contexts in which the word appears.
For words to be used in many different contexts,
they must carry little meaning. Prime examples of
such insignificant words are high-frequency stop
words, which are indeed represented by short vectors
despite their high term frequencies ...
For given term frequency,
the vector length is seen to take values only in a
narrow interval. That interval initially shifts upwards
with increasing frequency. Around a frequency
of about 30, that trend reverses and the interval
shifts downwards.
...
Both forces determining the length of a word
vector are seen at work here. Small-frequency
words tend to be used consistently, so that the
more frequently such words appear, the longer
their vectors. This tendency is reflected by the upwards
trend in Fig. 3 at low frequencies. High-frequency
words, on the other hand, tend to be
used in many different contexts, the more so, the
more frequently they occur. The averaging over
an increasing number of different contexts shortens
the vectors representing such words. This tendency
is clearly reflected by the downwards trend
in Fig. 3 at high frequencies, culminating in punctuation
marks and stop words with short vectors at
the very end.
...
Figure 3: Word vector length v versus term frequency
tf of all words in the hep-th vocabulary.
Note the logarithmic scale used on the frequency
axis. The dark symbols denote bin means with the
kth bin containing the frequencies in the interval
[2k−1, 2k − 1] with k = 1, 2, 3, . . .. These means
are included as a guide to the eye. The horizontal
line indicates the length v = 1.37 of the mean
vector
4 Discussion
Most applications of distributed representations of
words obtained through word2vec so far centered
around semantics. A host of experiments have
demonstrated the extent to which the direction of
word vectors captures semantics. In this brief report,
it was pointed out that not only the direction,
but also the length of word vectors carries important
information. Specifically, it was shown that
word vector length furnishes, in combination with
term frequency, a useful measure of word significance.
我通过 gensim 使用 Word2vec 和 Google 在 Google 新闻上训练的预训练向量。我注意到我可以通过对 Word2Vec 对象进行直接索引查找来访问的词向量不是单位向量：
>>> import numpy
>>> from gensim.models import Word2Vec
>>> w2v = Word2Vec.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
>>> king_vector = w2v['king']
>>> numpy.linalg.norm(king_vector)
2.9022589
但是，在most_similar方法中，没有使用这些非单位向量；相反，使用来自未记录的 .syn0norm 属性 的规范化版本，它仅包含单位向量：
>>> w2v.init_sims()
>>> unit_king_vector = w2v.syn0norm[w2v.vocab['king'].index]
>>> numpy.linalg.norm(unit_king_vector)
0.99999994
更大的向量只是单位向量的放大版本：
>>> king_vector - numpy.linalg.norm(king_vector) * unit_king_vector
array([ 0.00000000e+00, -1.86264515e-09, 0.00000000e+00,
0.00000000e+00, -1.86264515e-09, 0.00000000e+00,
-7.45058060e-09, 0.00000000e+00, 3.72529030e-09,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
... (some lines omitted) ...
-1.86264515e-09, -3.72529030e-09, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00], dtype=float32)
鉴于 Word2Vec 中的单词相似性比较是由 cosine similarity 完成的，对我来说非标准化向量的长度意味着什么并不明显 - 虽然我假设它们意味着 某事 ，因为 gensim 将它们暴露给我，而不是仅在 .syn0norm.
这些非规范化的Word2vec向量的长度是如何生成的，它们的含义是什么？对于哪些计算使用归一化向量有意义，什么时候应该使用非归一化向量？
我认为您正在寻找的答案在 2015 年的论文 Measuring Word Significance 中有所描述 使用 Adriaan Schakel 和 Benjamin Wilson 的分布式单词表示。关键点：
When a word appears in different contexts, its vector gets moved in different directions during updates. The final vector then represents some sort of weighted average over the various contexts. Averaging over vectors that point in different directions typically results in a vector that gets shorter with increasing number of different contexts in which the word appears. For words to be used in many different contexts, they must carry little meaning. Prime examples of such insignificant words are high-frequency stop words, which are indeed represented by short vectors despite their high term frequencies ...
For given term frequency, the vector length is seen to take values only in a narrow interval. That interval initially shifts upwards with increasing frequency. Around a frequency of about 30, that trend reverses and the interval shifts downwards.
...
Both forces determining the length of a word vector are seen at work here. Small-frequency words tend to be used consistently, so that the more frequently such words appear, the longer their vectors. This tendency is reflected by the upwards trend in Fig. 3 at low frequencies. High-frequency words, on the other hand, tend to be used in many different contexts, the more so, the more frequently they occur. The averaging over an increasing number of different contexts shortens the vectors representing such words. This tendency is clearly reflected by the downwards trend in Fig. 3 at high frequencies, culminating in punctuation marks and stop words with short vectors at the very end.
...
Figure 3: Word vector length v versus term frequency tf of all words in the hep-th vocabulary. Note the logarithmic scale used on the frequency axis. The dark symbols denote bin means with the kth bin containing the frequencies in the interval [2k−1, 2k − 1] with k = 1, 2, 3, . . .. These means are included as a guide to the eye. The horizontal line indicates the length v = 1.37 of the mean vector
4 Discussion
Most applications of distributed representations of words obtained through word2vec so far centered around semantics. A host of experiments have demonstrated the extent to which the direction of word vectors captures semantics. In this brief report, it was pointed out that not only the direction, but also the length of word vectors carries important information. Specifically, it was shown that word vector length furnishes, in combination with term frequency, a useful measure of word significance.