使用quanteda计算term specific term和inverse term frq
Use quanteda to calculate term specific term and inverse term frq
示例测试数据集:
library(quanteda)
dataset1 <- data.frame( anumber = c(1,2,3), text = c("Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.","It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum", "Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source."))
myDfm <- dataset1 %>%
corpus() %>%
tokens(remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE) %>%
tokens_ngrams(n = 1:3) %>%
dfm()
如何计算所有文档的 tf,并将其乘以特定于术语的 idf,并将结果再次作为 dfm?
这里的诀窍是计算所有文档中组合的特征(术语)频率(通常在文档中计算),并将其乘以文档频率(始终按特征计算,跨所有文档)。
然后您可以为整个集合计算每个特征的“tf-idf”分数。您使用两个适用于 dfm 对象的函数来执行此操作:featfreq()
用于 term/feature 频率,docfreq()
用于文档频率。
library(quanteda)
## Package version: 2.1.1
# your tokenization and dfm code
myDfm <- dataset1 %>%
corpus() %>%
tokens(remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE) %>%
tokens_ngrams(n = 1:3) %>%
dfm()
这会计算整个语料库中每个特征的 tf-idf,与文档中每个特征的 dfm_tfidf()
相同。它还按降序对它们进行排序。
result <- featfreq(myDfm) * log(ndoc(myDfm) / docfreq(myDfm), base = 10) %>%
sort(decreasing = TRUE)
这些是您的热门条款:
head(result, 10)
## lorem ipsum is simply dummy text of the
## 2.8627275 2.8627275 0.9542425 0.9542425 0.9542425 1.4313638 3.3398488 4.7712125
## printing and
## 0.4771213 1.9084850
还有你的最低条件:
tail(result, 10)
## the_cites_of cites_of_the
## 0.1760913 0.1760913
## of_the_word the_word_in
## 0.0000000 0.0000000
## word_in_classical in_classical_literature
## 0.0000000 0.0000000
## classical_literature_discovered literature_discovered_the
## 0.0000000 0.0000000
## discovered_the_undoubtable the_undoubtable_source
## 0.0000000 0.0000000
示例测试数据集:
library(quanteda)
dataset1 <- data.frame( anumber = c(1,2,3), text = c("Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.","It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum", "Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source."))
myDfm <- dataset1 %>%
corpus() %>%
tokens(remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE) %>%
tokens_ngrams(n = 1:3) %>%
dfm()
如何计算所有文档的 tf,并将其乘以特定于术语的 idf,并将结果再次作为 dfm?
这里的诀窍是计算所有文档中组合的特征(术语)频率(通常在文档中计算),并将其乘以文档频率(始终按特征计算,跨所有文档)。
然后您可以为整个集合计算每个特征的“tf-idf”分数。您使用两个适用于 dfm 对象的函数来执行此操作:featfreq()
用于 term/feature 频率,docfreq()
用于文档频率。
library(quanteda)
## Package version: 2.1.1
# your tokenization and dfm code
myDfm <- dataset1 %>%
corpus() %>%
tokens(remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE) %>%
tokens_ngrams(n = 1:3) %>%
dfm()
这会计算整个语料库中每个特征的 tf-idf,与文档中每个特征的 dfm_tfidf()
相同。它还按降序对它们进行排序。
result <- featfreq(myDfm) * log(ndoc(myDfm) / docfreq(myDfm), base = 10) %>%
sort(decreasing = TRUE)
这些是您的热门条款:
head(result, 10)
## lorem ipsum is simply dummy text of the
## 2.8627275 2.8627275 0.9542425 0.9542425 0.9542425 1.4313638 3.3398488 4.7712125
## printing and
## 0.4771213 1.9084850
还有你的最低条件:
tail(result, 10)
## the_cites_of cites_of_the
## 0.1760913 0.1760913
## of_the_word the_word_in
## 0.0000000 0.0000000
## word_in_classical in_classical_literature
## 0.0000000 0.0000000
## classical_literature_discovered literature_discovered_the
## 0.0000000 0.0000000
## discovered_the_undoubtable the_undoubtable_source
## 0.0000000 0.0000000