如何使用整齐的文本使用二元组和三元组

Question

我正在尝试使用 tidytext 同时使用二元语法和三元语法。我可以为令牌使用什么代码来查找 2 和 3 个单词。

这是仅使用双字母组的代码：

library(tidytext)
library(janeaustenr)

austen_bigrams <- austen_books() %>%
  unnest_tokens(bigram, text, token = "ngrams", n = 2)

austen_bigrams

Answer 1

如果您查看 ?unnest_tokens，它会告诉您 ... 用于传递给分词器的参数。对于 ngrams，那是 tokenizers::tokenize_ngrams，如果你查看它的帮助页面，它有一个 n_min 参数，所以你可以做

library(magrittr)
library(tidytext)
library(janeaustenr)

austen_bigrams <- austen_books() %>% 
    head(1000) %>%    # otherwise this will get very large
    unnest_tokens(bigram, text, token = "ngrams", n = 3, n_min = 2)

austen_bigrams
#> # A tibble: 19,801 x 2
#>                   book                bigram
#>                 <fctr>                 <chr>
#>  1 Sense & Sensibility             sense and
#>  2 Sense & Sensibility sense and sensibility
#>  3 Sense & Sensibility       and sensibility
#>  4 Sense & Sensibility    and sensibility by
#>  5 Sense & Sensibility        sensibility by
#>  6 Sense & Sensibility   sensibility by jane
#>  7 Sense & Sensibility               by jane
#>  8 Sense & Sensibility        by jane austen
#>  9 Sense & Sensibility           jane austen
#> 10 Sense & Sensibility      jane austen 1811
#> # ... with 19,791 more rows

如何使用整齐的文本使用二元组和三元组

How to use bigrams and trigrams using tidy text

r

token

tidytext