从 dtm 中删除单词
Remove words from a dtm
我创建了一个 dtm。
library(tm)
corpus = Corpus(VectorSource(dat$Reviews))
dtm = DocumentTermMatrix(corpus)
我用它来删除稀有词。
dtm = removeSparseTerms(dtm, 0.98)
在removeSparseTerms之后dtm中还有一些术语对我的分析没有用。
tm包有去词功能。但是，此功能只能应用于语料库或向量。
如何从 dtm 中删除已定义的术语？
这是输入数据的一个小样本：
samp = dat %>%
select(Reviews) %>%
sample_n(20)
dput(samp)
structure(list(Reviews = c("buenisimoooooo", "excelente", "excelent",
"awesome phone awesome price almost month issue highly use blu manufacturer high speed processor blu iphone",
"phone multiple failure poorly touch screen 2 slot sim card work responsible disappoint brand good team shop store wine money unfortunately precaution purchase",
"work perfect time", "amaze buy phone smoothly update charm glte yet comparably fast several different provider sims perfectly small size definitely replacemnent simple",
"phone work card non sim card description", "perfect reliable kinda fast even simple mobile sim digicel never problem far strongly anyone need nice expensive dual sim phone perfect gift love friend",
"perfect", "great bang buck", "actually happy little sister really first good great picture late",
"good phone good reception home fringe area screen lovely just right size good buy",
"", "phone verizon contract phone buyer beware", "good phone",
"excellent product total satisfaction", "dreadful phone home button never screen unresponsive answer call easily month phone test automatically emergency police round supplier network nothing never electricals amazon good buy locally refund",
"good phone price fine", "phone star battery little soon yes"
)), row.names = c(12647L, 10088L, 14055L, 3720L, 6588L, 10626L,
10362L, 1428L, 12580L, 5381L, 10431L, 2803L, 6644L, 12969L, 348L,
10582L, 3215L, 13358L, 12708L, 7049L), class = "data.frame")
您应该尝试 quanteda，它调用 DocumentTermMatrix "dfm"（文档特征矩阵）并且有更多选项 trim 它可以减少稀疏性，包括用于删除特定特征（术语）的函数 dfm_remove()。
如果我们将您的 samp 对象重命名为 dat，那么：
library("quanteda")
## Package version: 1.4.3
## Parallel computing: 2 of 12 threads used.
## See https://quanteda.io for tutorials and examples.
corp <- corpus(dat, text_field = "Reviews")
corp
## Corpus consisting of 20 documents and 0 docvars.
tail(texts(corp), 2)
## 12708 7049
## "good phone price fine" "phone star battery little soon yes"
dtm <- dfm(corp)
dtm
## Document-feature matrix of: 20 documents, 128 features (93.6% sparse).
现在我们可以trim了。对于这个小的，0.98的稀疏度设置没有影响，但是我们可以trim基于频率阈值。
# does not actually have an effect
dfm_trim(dtm, sparsity = 0.98, verbose = TRUE)
## Note: converting sparsity into min_docfreq = 1 - 0.98 = NULL .
## No features removed.
## Document-feature matrix of: 20 documents, 128 features (93.6% sparse).
# trimming based on rare terms
dtm <- dfm_trim(dtm, min_termfreq = 3, verbose = TRUE)
## Removing features occurring:
## - fewer than 3 times: 119
## Total features removed: 119 (93.0%).
head(dtm)
## Document-feature matrix of: 6 documents, 9 features (83.3% sparse).
## 6 x 9 sparse Matrix of class "dfm"
## features
## docs phone screen sim card work good perfect buy never
## 12647 0 0 0 0 0 0 0 0 0
## 10088 0 0 0 0 0 0 0 0 0
## 14055 0 0 0 0 0 0 0 0 0
## 3720 1 0 0 0 0 0 0 0 0
## 6588 1 1 1 1 1 1 0 0 0
## 10626 0 0 0 0 1 0 1 0 0
无论如何直接回答您的问题，您希望dfm_remove()摆脱特定功能。
# removing from a specific list of terms
dtm <- dfm_remove(dtm, c("screen", "buy", "sim", "card"), verbose = TRUE)
## removed 4 features
##
dtm
## Document-feature matrix of: 20 documents, 5 features (75.0% sparse).
head(dtm)
## Document-feature matrix of: 6 documents, 5 features (80.0% sparse).
## 6 x 5 sparse Matrix of class "dfm"
## features
## docs phone work good perfect never
## 12647 0 0 0 0 0
## 10088 0 0 0 0 0
## 14055 0 0 0 0 0
## 3720 1 0 0 0 0
## 6588 1 1 1 0 0
## 10626 0 1 0 1 0
最后，如果您仍然想要，可以使用 quanteda 将 dtm 转换为 tm 格式的 convert() 函数：
convert(dtm, to = "tm")
## <<DocumentTermMatrix (documents: 20, terms: 5)>>
## Non-/sparse entries: 25/75
## Sparsity : 75%
## Maximal term length: 7
## Weighting : term frequency (tf)
我创建了一个 dtm。
library(tm)
corpus = Corpus(VectorSource(dat$Reviews))
dtm = DocumentTermMatrix(corpus)
我用它来删除稀有词。
dtm = removeSparseTerms(dtm, 0.98)
在removeSparseTerms之后dtm中还有一些术语对我的分析没有用。
tm包有去词功能。但是，此功能只能应用于语料库或向量。
如何从 dtm 中删除已定义的术语？
这是输入数据的一个小样本：
samp = dat %>%
select(Reviews) %>%
sample_n(20)
dput(samp)
structure(list(Reviews = c("buenisimoooooo", "excelente", "excelent",
"awesome phone awesome price almost month issue highly use blu manufacturer high speed processor blu iphone",
"phone multiple failure poorly touch screen 2 slot sim card work responsible disappoint brand good team shop store wine money unfortunately precaution purchase",
"work perfect time", "amaze buy phone smoothly update charm glte yet comparably fast several different provider sims perfectly small size definitely replacemnent simple",
"phone work card non sim card description", "perfect reliable kinda fast even simple mobile sim digicel never problem far strongly anyone need nice expensive dual sim phone perfect gift love friend",
"perfect", "great bang buck", "actually happy little sister really first good great picture late",
"good phone good reception home fringe area screen lovely just right size good buy",
"", "phone verizon contract phone buyer beware", "good phone",
"excellent product total satisfaction", "dreadful phone home button never screen unresponsive answer call easily month phone test automatically emergency police round supplier network nothing never electricals amazon good buy locally refund",
"good phone price fine", "phone star battery little soon yes"
)), row.names = c(12647L, 10088L, 14055L, 3720L, 6588L, 10626L,
10362L, 1428L, 12580L, 5381L, 10431L, 2803L, 6644L, 12969L, 348L,
10582L, 3215L, 13358L, 12708L, 7049L), class = "data.frame")
您应该尝试 quanteda，它调用 DocumentTermMatrix "dfm"（文档特征矩阵）并且有更多选项 trim 它可以减少稀疏性，包括用于删除特定特征（术语）的函数 dfm_remove()。
如果我们将您的 samp 对象重命名为 dat，那么：
library("quanteda")
## Package version: 1.4.3
## Parallel computing: 2 of 12 threads used.
## See https://quanteda.io for tutorials and examples.
corp <- corpus(dat, text_field = "Reviews")
corp
## Corpus consisting of 20 documents and 0 docvars.
tail(texts(corp), 2)
## 12708 7049
## "good phone price fine" "phone star battery little soon yes"
dtm <- dfm(corp)
dtm
## Document-feature matrix of: 20 documents, 128 features (93.6% sparse).
现在我们可以trim了。对于这个小的，0.98的稀疏度设置没有影响，但是我们可以trim基于频率阈值。
# does not actually have an effect
dfm_trim(dtm, sparsity = 0.98, verbose = TRUE)
## Note: converting sparsity into min_docfreq = 1 - 0.98 = NULL .
## No features removed.
## Document-feature matrix of: 20 documents, 128 features (93.6% sparse).
# trimming based on rare terms
dtm <- dfm_trim(dtm, min_termfreq = 3, verbose = TRUE)
## Removing features occurring:
## - fewer than 3 times: 119
## Total features removed: 119 (93.0%).
head(dtm)
## Document-feature matrix of: 6 documents, 9 features (83.3% sparse).
## 6 x 9 sparse Matrix of class "dfm"
## features
## docs phone screen sim card work good perfect buy never
## 12647 0 0 0 0 0 0 0 0 0
## 10088 0 0 0 0 0 0 0 0 0
## 14055 0 0 0 0 0 0 0 0 0
## 3720 1 0 0 0 0 0 0 0 0
## 6588 1 1 1 1 1 1 0 0 0
## 10626 0 0 0 0 1 0 1 0 0
无论如何直接回答您的问题，您希望dfm_remove()摆脱特定功能。
# removing from a specific list of terms
dtm <- dfm_remove(dtm, c("screen", "buy", "sim", "card"), verbose = TRUE)
## removed 4 features
##
dtm
## Document-feature matrix of: 20 documents, 5 features (75.0% sparse).
head(dtm)
## Document-feature matrix of: 6 documents, 5 features (80.0% sparse).
## 6 x 5 sparse Matrix of class "dfm"
## features
## docs phone work good perfect never
## 12647 0 0 0 0 0
## 10088 0 0 0 0 0
## 14055 0 0 0 0 0
## 3720 1 0 0 0 0
## 6588 1 1 1 0 0
## 10626 0 1 0 1 0
最后，如果您仍然想要，可以使用 quanteda 将 dtm 转换为 tm 格式的 convert() 函数：
convert(dtm, to = "tm")
## <<DocumentTermMatrix (documents: 20, terms: 5)>>
## Non-/sparse entries: 25/75
## Sparsity : 75%
## Maximal term length: 7
## Weighting : term frequency (tf)