如何在 tidytext 中包含 select 个 2 词短语作为标记？

Question

我正在预处理一些文本数据以供进一步分析。我使用 unnest_tokens() [将文本标记为单数]，但想保留某些经常出现的 2 个单词短语，例如 "United States" 或 "social security." 我如何使用 tidytext 做到这一点？

tidy_data <- data %>%
                unnest_tokens(word, text) %>%
                anti_join(stop_words)

dput(data[1:6, 1:6])

structure(list(race = c("US House", "US House", "US House", "US House", 
"", "US House"), district = c(8L, 3L, 6L, 17L, 2L, 1L), party = c("Republican", 
"Republican", "Republican", "Republican", "", "Republican"), 
    state = c("AZ", "AZ", "KY", "TX", "IL", "NH"), sponsor = c(4, 
    4, 4, 1, NA, 4), approve = structure(c(1L, 1L, 1L, 4L, NA, 
    1L), .Label = c("no oral statement of approval, authorization", 
    "beginning of the spot", "middle of the spot", "end of the spot"
    ), class = "factor")), row.names = c(NA, 6L), class = "data.frame")

Answer 1

如果我处于这种情况并且我只有一个简短的双词短语列表需要保留在我的分析中，我会在标记化之前和之后进行一些谨慎的替换。

首先，我会将双词短语替换为可以粘在一起并且不会被我正在使用的标记化过程分解的东西，比如 "united states" 到 "united_states"。

library(tidyverse)
library(tidytext)


df <- tibble(text = c("I live in the United States",
                      "United we stand, divided we fall",
                      "Information security is important!",
                      "I work at the Social Security Administration"))


df_parsed <- df %>%
  mutate(text = str_to_lower(text),
         text = str_replace_all(text, "united states", "united_states"),
         text = str_replace_all(text, "social security", "social_security"))

df_parsed
#> # A tibble: 4 x 1
#>   text                                        
#>   <chr>                                       
#> 1 i live in the united_states                 
#> 2 united we stand, divided we fall            
#> 3 information security is important!          
#> 4 i work at the social_security administration

然后你就可以正常分词了，然后再把你刚刚做的东西换成双词短语，所以"united_states"回到"united states"。

df_parsed %>%
  unnest_tokens(word, text) %>%
  mutate(word = case_when(word == "united_states" ~ "united states",
                          word == "social_security" ~ "social security",
                          TRUE ~ word))
#> # A tibble: 21 x 1
#>    word         
#>    <chr>        
#>  1 i            
#>  2 live         
#>  3 in           
#>  4 the          
#>  5 united states
#>  6 united       
#>  7 we           
#>  8 stand        
#>  9 divided      
#> 10 we           
#> # … with 11 more rows

^{由 reprex package (v0.3.0)}

于 2019-08-03 创建

如果你有一个长长的列表，它会变得困难和繁重，然后看看使用 bigram 和 unigram 标记化的方法可能是有意义的。您可以看到 here.

的示例

如何在 tidytext 中包含 select 个 2 词短语作为标记？

How to include select 2-word phrases as tokens in tidytext?

r

tokenize

tidytext