语料库包的 tolower 函数抛出错误

Question

我正在尝试使用 Twitter 数据进行一些文本挖掘。我执行以下操作：

#connect to twitter API
setup_twitter_oauth(consumer_key, consumer_secret, access_token, access_secret)

#set radius and amount of requests
N=200  # tweets to request from each query
S=200  # radius in miles

lats=c(38.9,40.7)
lons=c(-77,-74)

roger=do.call(rbind,lapply(1:length(lats), function(i) searchTwitter('Roger+Federer',
                                                                  lang="en",n=N,resultType="recent",
                                                                  geocode=paste(lats[i],lons[i],paste0(S,"mi"),sep=","))))

一切正常，但是当我想像这样使用 corpus 包的 tolower 函数时：

data=as.data.frame(cbind(tweet=rogertext))
corpus=Corpus(VectorSource(data$tweet))
corpus=tm_map(corpus,tolower)

它抛出了这个错误：

> corpus=tm_map(corpus,tolower)
Error in FUN(X[[i]], ...) : 
invalid input 'RT @Federerism: Roger Federer reaches  5 million followers   on twitter  Love You Roger í ½í¸˜ í ½í¸ í ½í¸˜ í ½í¸ #Roger #Federer #   Federerism #Maestro https:/â€¦' in 'utf8towcs'

想过哪里出了问题吗？

Answer 1

尝试以下操作：

corpus <- tm_map(corpus, content_transformer(tolower))

几年前 tm 包中的语法发生了变化。希望这能解决问题。

Answer 2

base::tolower 遇到特殊字符会卡住。在挖掘推文时，这通常是一个问题。您可以尝试捕获错误或只使用 stringi 的 tolower pendant：

# tw <- searchTwitter('Roger Federer reaches  5 million followers   on twitter  Love You Roger', n=1) 
download.file("https://www.dropbox.com/s/33ilhcu2v82nwuq/twitter_tolower.rda?dl=1", tf <- tempfile(fileext = ".rda"), mode="wb")
load(tf) 

tw[[1]]$getText()
# [1] "RT @Federerism: Roger Federer reaches  5 million followers on twitter  Love You Roger \xed��\xed�\u0098 \xed��\xed�\u008d \xed��\xed�\u0098 \xed��\xed�\u008d #Roger #Federer # Federerism #Maestro https:/…"

## Does not work:
tolower(tw[[1]]$getText())
# Error in tolower(tw[[1]]$getText()) : 
#   invalid input 'RT @Federerism: Roger Federer reaches  5 million followers on twitter  Love You Roger í ½í¸˜ í ½í¸ í ½í¸˜ í ½í¸ #Roger #Federer # Federerism #Maestro https:/â€¦' in 'utf8towcs'

## Works:
stringi::stri_trans_tolower(tw[[1]]$getText())
# [1] "rt @federerism: roger federer reaches  5 million followers on twitter  love you roger \xed��\xed�\u0098 \xed��\xed�\u008d \xed��\xed�\u0098 \xed��\xed�\u008d #roger #federer # federerism #maestro https:/…"

## Works, too:
library(tm)
corp <- Corpus(VectorSource(tw[[1]]$getText()))
corp <- tm_map(corp, content_transformer(stringi::stri_trans_tolower))
content(corp[[1]])
# [1] "rt @federerism: roger federer reaches  5 million followers on twitter  love you roger \xed��\xed�\u0098 \xed��\xed�\u008d \xed��\xed�\u0098 \xed��\xed�\u008d #roger #federer # federerism #maestro https:/…"

语料库包的 tolower 函数抛出错误

tolower function of the corpus package throws an error

r

text-mining

tm

corpus