Quanteda 软件包，朴素贝叶斯：我如何预测不同特征的测试数据？

Question

我使用 quanteda::textmodel_NB 创建了一个模型，将文本分为两个类别之一。我将模型拟合到去年夏天的训练数据集上。

现在，今年夏天我正尝试使用它来对我们在工作中收到的新文本进行分类。我尝试这样做并收到以下错误：

Error in predict.textmodel_NB_fitted(model, test_dfm) : 
feature set in newdata different from that in training set

函数中产生错误的代码can be found here at lines 157 to 165.

我假设发生这种情况是因为训练数据集中的词与测试数据集中使用的词不完全匹配。但是为什么会出现这个错误呢？我觉得好像——为了在现实世界的例子中有用——模型应该能够处理包含不同特征的数据集，因为这在应用使用中可能总是会发生。

所以我的第一个问题是：

1.这个错误是朴素贝叶斯算法的属性吗？还是函数的作者选择这样做？

这引出了我的第二个问题：

2。我该如何解决这个问题？

为了解决第二个问题，我提供了可重现的代码（最后一行生成了上面的错误）：

library(quanteda)
library(magrittr)
library(data.table)

train_text <- c("Can random effects apply only to categorical variables?",
                "ANOVA expectation identity",
                "Statistical test for significance in ranking positions",
                "Is Fisher Sharp Null Hypothesis testable?",
                "List major reasons for different results from survival analysis among different studies",
                "How do the tenses and aspects in English correspond temporally to one another?",
                "Is there a correct gender-neutral singular pronoun (“his” vs. “her” vs. “their”)?",
                "Are collective nouns always plural, or are certain ones singular?",
                "What’s the rule for using “who” and “whom” correctly?",
                "When is a gerund supposed to be preceded by a possessive adjective/determiner?")

train_class <- factor(c(rep(0,5), rep(1,5)))

train_dfm <- train_text %>% 
  dfm(tolower=TRUE, stem=TRUE, remove=stopwords("english"))

model <- textmodel_NB(train_dfm, train_class)

test_text <- c("Weighted Linear Regression with Proportional Standard Deviations in R",
               "What do significance tests for adjusted means tell us?",
               "How should I punctuate around quotes?",
               "Should I put a comma before the last item in a list?")

test_dfm <- test_text %>% 
  dfm(tolower=TRUE, stem=TRUE, remove=stopwords("english"))

predict(model, test_dfm)

我唯一想做的就是手动使特征相同（我假设这会为对象中不存在的特征填充 0），但这生成了一个新的错误。上面例子的代码是：

model_features <- model$data$x@Dimnames$features # gets the features of the training data

test_features <- test_dfm@Dimnames$features # gets the features of the test data

all_features <- c(model_features, test_features) %>% # combining the two sets of features...
  subset(!duplicated(.)) # ...and getting rid of duplicate features

model$data$x@Dimnames$features <- test_dfm@Dimnames$features <- all_features # replacing features of model and test_dfm with all_features

predict(model, dfm) # new error?

但是，此代码会生成 new 错误：

Error in if (ncol(object$PcGw) != ncol(newdata)) stop("feature set in newdata different from that in training set") : 
  argument is of length zero

如何将这个朴素贝叶斯模型应用于具有不同特征的新数据集？

Answer 1

幸运的是，有一种简单的方法可以做到这一点：您可以在测试数据上使用 dfm_select() 来为训练集提供相同的特征（以及特征的排序）。就这么简单:

test_dfm <- dfm_select(test_dfm, train_dfm)
predict(model, test_dfm)
## Predicted textmodel of type: Naive Bayes
## 
##             lp(0)       lp(1)     Pr(0)  Pr(1) Predicted
## text1  -0.6931472  -0.6931472    0.5000 0.5000         0
## text2 -11.8698712 -13.1879095    0.7889 0.2111         0
## text3  -4.1484118  -3.6635616    0.3811 0.6189         1
## text4  -8.0091415  -8.4257356    0.6027 0.3973         0

Answer 2

截至 2018 年 5 月，现在似乎有一个 "force = TRUE" 选项也可以为您完成这项工作：

predict(model, test_dfm, force = TRUE)
# text1 text2 text3 text4 
#    0     0     1     0 
# Levels: 0 1

来源：koheiw 和 kbenoit 关于 quanteda 的讨论 Github - https://github.com/quanteda/quanteda/issues/1329

Quanteda 软件包，朴素贝叶斯：我如何预测不同特征的测试数据？

Quanteda package, Naive Bayes: How can I predict on different-featured test data?

r

text-analysis

naivebayes

quanteda