dtm 稀疏性不同取决于 tf/tfidf ，相同的语料库
dtm sparsity different depending on tf/tfidf , same corpus
谁能解释一下？
我的理解：
tf >= 0 (absolute frequency value)
tfidf >= 0 (for negative idf, tf=0)
sparse entry = 0
nonsparse entry > 0
因此，在使用以下代码创建的两个 DTM 中，确切的 sparse/nonsparse 比例应该相同。
library(tm)
data(crude)
dtm <- DocumentTermMatrix(crude, control=list(weighting=weightTf))
dtm2 <- DocumentTermMatrix(crude, control=list(weighting=weightTfIdf))
dtm
dtm2
但是：
> dtm
<<DocumentTermMatrix (documents: 20, terms: 1266)>>
**Non-/sparse entries: 2255/23065**
Sparsity : 91%
Maximal term length: 17
Weighting : term frequency (tf)
> dtm2
<<DocumentTermMatrix (documents: 20, terms: 1266)>>
**Non-/sparse entries: 2215/23105**
Sparsity : 91%
Maximal term length: 17
Weighting : term frequency - inverse document frequency (normalized) (tf-idf)
稀疏度可以不同。如果 TF 为零或 IDF 为零，则 TF-IDF 值将为零，如果每个文档中都出现一个术语，则 IDF 为零。考虑以下示例：
txts <- c("super World", "Hello World", "Hello super top world")
library(tm)
tf <- TermDocumentMatrix(Corpus(VectorSource(txts)), control=list(weighting=weightTf))
tfidf <- TermDocumentMatrix(Corpus(VectorSource(txts)), control=list(weighting=weightTfIdf))
inspect(tf)
# <<TermDocumentMatrix (terms: 4, documents: 3)>>
# Non-/sparse entries: 8/4
# Sparsity : 33%
# Maximal term length: 5
# Weighting : term frequency (tf)
#
# Docs
# Terms 1 2 3
# hello 0 1 1
# super 1 0 1
# top 0 0 1
# world 1 1 1
inspect(tfidf)
# <<TermDocumentMatrix (terms: 4, documents: 3)>>
# Non-/sparse entries: 5/7
# Sparsity : 58%
# Maximal term length: 5
# Weighting : term frequency - inverse document frequency (normalized) (tf-idf)
#
# Docs
# Terms 1 2 3
# hello 0.0000000 0.2924813 0.1462406
# super 0.2924813 0.0000000 0.1462406
# top 0.0000000 0.0000000 0.3962406
# world 0.0000000 0.0000000 0.0000000
术语 super 在文档 1 中出现 1 次，文档 1 有 2 个术语，它出现在 3 个文档中的 2 个中：
1/2 * log2(3/2)
# [1] 0.2924813
术语 world 在文档 3 中出现 1 次，文档 3 有 4 个术语，并且出现在所有 3 个文档中：
1/4 * log2(3/3) # 1/4 * 0
# [1] 0
谁能解释一下？
我的理解：
tf >= 0 (absolute frequency value)
tfidf >= 0 (for negative idf, tf=0)
sparse entry = 0
nonsparse entry > 0
因此，在使用以下代码创建的两个 DTM 中，确切的 sparse/nonsparse 比例应该相同。
library(tm)
data(crude)
dtm <- DocumentTermMatrix(crude, control=list(weighting=weightTf))
dtm2 <- DocumentTermMatrix(crude, control=list(weighting=weightTfIdf))
dtm
dtm2
但是：
> dtm
<<DocumentTermMatrix (documents: 20, terms: 1266)>>
**Non-/sparse entries: 2255/23065**
Sparsity : 91%
Maximal term length: 17
Weighting : term frequency (tf)
> dtm2
<<DocumentTermMatrix (documents: 20, terms: 1266)>>
**Non-/sparse entries: 2215/23105**
Sparsity : 91%
Maximal term length: 17
Weighting : term frequency - inverse document frequency (normalized) (tf-idf)
稀疏度可以不同。如果 TF 为零或 IDF 为零，则 TF-IDF 值将为零，如果每个文档中都出现一个术语，则 IDF 为零。考虑以下示例：
txts <- c("super World", "Hello World", "Hello super top world")
library(tm)
tf <- TermDocumentMatrix(Corpus(VectorSource(txts)), control=list(weighting=weightTf))
tfidf <- TermDocumentMatrix(Corpus(VectorSource(txts)), control=list(weighting=weightTfIdf))
inspect(tf)
# <<TermDocumentMatrix (terms: 4, documents: 3)>>
# Non-/sparse entries: 8/4
# Sparsity : 33%
# Maximal term length: 5
# Weighting : term frequency (tf)
#
# Docs
# Terms 1 2 3
# hello 0 1 1
# super 1 0 1
# top 0 0 1
# world 1 1 1
inspect(tfidf)
# <<TermDocumentMatrix (terms: 4, documents: 3)>>
# Non-/sparse entries: 5/7
# Sparsity : 58%
# Maximal term length: 5
# Weighting : term frequency - inverse document frequency (normalized) (tf-idf)
#
# Docs
# Terms 1 2 3
# hello 0.0000000 0.2924813 0.1462406
# super 0.2924813 0.0000000 0.1462406
# top 0.0000000 0.0000000 0.3962406
# world 0.0000000 0.0000000 0.0000000
术语 super 在文档 1 中出现 1 次，文档 1 有 2 个术语，它出现在 3 个文档中的 2 个中：
1/2 * log2(3/2)
# [1] 0.2924813
术语 world 在文档 3 中出现 1 次，文档 3 有 4 个术语，并且出现在所有 3 个文档中：
1/4 * log2(3/3) # 1/4 * 0
# [1] 0