将主题模型输出转换为 JSON

Question

我使用以下函数将 topicmodels 输出转换为 JSON 输出以在 ldavis 中使用。

topicmodels_json_ldavis <- function(fitted, corpus, doc_term){
     ## Required packages
     library(topicmodels)
     library(dplyr)
     library(stringi)
     library(tm)
     library(LDAvis)

     ## Find required quantities
     phi <- posterior(fitted)$terms %>% as.matrix
     theta <- posterior(fitted)$topics %>% as.matrix
     vocab <- colnames(phi)
     doc_length <- vector()
     for (i in 1:length(corpus)) {
          temp <- paste(corpus[[i]]$content, collapse = ' ')
          doc_length <- c(doc_length, stri_count(temp, regex = '\S+'))
     }
     temp_frequency <- inspect(doc_term)
     freq_matrix <- data.frame(ST = colnames(temp_frequency),
                               Freq = colSums(temp_frequency))
     rm(temp_frequency)

     ## Convert to json
     json_lda <- LDAvis::createJSON(phi = phi, theta = theta,
                                    vocab = vocab,
                                    doc.length = doc_length,
                                    term.frequency = freq_matrix$Freq)

     return(json_lda)
}

但我收到以下错误

错误 LDAvis::createJSON(phi = phi, theta = theta, vocab = vocab, doc.length = doc_length, : doc.length 的长度不等于到 theta 中的行数；两者都应该等于的数量数据中的文档。

这是我的完整代码：

data <- read.csv("textmining.csv")


corpus <- Corpus(DataframeSource(data.frame(data$reasonforleaving))) 

# Remove punctuations and numbers because they are generally uninformative.
corpus <- tm_map(corpus, removePunctuation)
corpus <- tm_map(corpus, removeNumbers)
# Convert all words to lowercase.
corpus <- tm_map(corpus, content_transformer(tolower))
# Remove stopwords such as "a", "the", etc.
corpus <- tm_map(corpus, removeWords, stopwords("english"))
# Use the SnowballC package to do stemming.
library(SnowballC)
corpus <- tm_map(corpus, stemDocument)


# remove extra words
toSpace <- content_transformer(function (x , pattern ) gsub(pattern, " ", x))
corpus <- tm_map(corpus, toSpace, "still")
corpus <- tm_map(corpus, toSpace, "also")

# Remove excess white spaces between words.

corpus <- tm_map(corpus, stripWhitespace)
# Inspect the first document to see what it looks like.
corpus[[1]]$content 

dtm <- DocumentTermMatrix(corpus)

# remove empty documents
library(slam)
dtm = dtm[row_sums(dtm)>0,]

# Use topicmodels package to conduct LDA analysis.

burnin <- 500
iter <- 1000
keep <- 30
k <- 5

result55 <- LDA(dtm, 5)
ldaoutput = topicmodels_json_ldavis(result55,corpus, dtm)

你知道我为什么会收到错误消息吗？

谢谢

Answer 1

问题

你的问题在for (i in 1:length(corpus))在

 doc_length <- vector()
     for (i in 1:length(corpus)) {
          temp <- paste(corpus[[i]]$content, collapse = ' ')
          doc_length <- c(doc_length, stri_count(temp, regex = '\S+'))
     }

请记住，您已从 dtm = dtm[row_sums(dtm)>0,] 中的 DocumentTermMatrix 中删除了一些 "empty" 个文档，所以你这里的矢量长度会太大。

建议

您可能希望保留空文档的向量，因为它不仅可以帮助您生成 JSON，还可以在空文档集和完整文档集之间来回切换。
doc.length = colSums( as.matrix(tdm) > 0 )[!empty.docs]

我的建议假设您拥有完整的 tdm 和空文档

Answer 2

我用相同的代码遇到了同样的问题，并找到了这个函数 here :

topicmodels2LDAvis <- function(x, ...){
    post <- topicmodels::posterior(x)
    if (ncol(post[["topics"]]) < 3) stop("The model must contain > 2 topics")
    mat <- x@wordassignments
    LDAvis::createJSON(
        phi = post[["terms"]], 
        theta = post[["topics"]],
        vocab = colnames(post[["terms"]]),
        doc.length = slam::row_sums(mat, na.rm = TRUE),
        term.frequency = slam::col_sums(mat, na.rm = TRUE)
    )
}

使用起来简单得多，只需将您的 LDA 结果作为参数：

result55 <- LDA(dtm, 5)
serVis(topicmodels2LDAvis(result55))

将主题模型输出转换为 JSON

Convert topicmodels output to JSON

text-mining

lda

topic-modeling

问题

建议