FastText：无法获取 cross_validation

Question

我正在努力将 FastText (FTTransformer) 实现到迭代不同矢量化器的管道中。更具体地说，我无法获得交叉验证分数。使用以下代码：

%%time
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from gensim.utils import simple_preprocess
from gensim.sklearn_api.ftmodel import FTTransformer
np.random.seed(0)

data = pd.read_csv('https://pastebin.com/raw/dqKFZ12m')
X_train, X_test, y_train, y_test = train_test_split(data.text, data.label, random_state=0)
w2v_texts = [simple_preprocess(doc) for doc in X_train]

models = [FTTransformer(size=10, min_count=0, seed=42)]
classifiers = [LogisticRegression(random_state=0)]

for model in models:

    for classifier in classifiers:

        model.fit(w2v_texts)
        classifier.fit(model.transform(X_train), y_train)

        pipeline = Pipeline([
                ('vec', model),
                ('clf', classifier)
            ])

        print(pipeline.score(X_train, y_train))
        #print(model.gensim_model.wv.most_similar('kirk'))

        cross_val_score(pipeline, X_train, y_train, scoring='accuracy', cv=5)

KeyError: 'all ngrams for word "Machine learning can be useful branding sometimes" absent from model'

问题如何解决？

旁注：我的其他带有 D2VTransformer 或 TfIdfVectorizer 的管道工作得很好。在这里，我可以在定义管道后简单地应用 pipeline.fit(X_train, y_train)，而不是如上所示的两个拟合。似乎 FTTransformer 不能很好地与其他给定的向量化器集成？

Answer 1

是的，要在管道中使用，需要修改 FTTransformer 以在其 fit 方法中将文档拆分为单词。可以这样做：

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from gensim.utils import simple_preprocess
from gensim.sklearn_api.ftmodel import FTTransformer
np.random.seed(0)


class FTTransformer2(FTTransformer):

    def fit(self, x, y):
        super().fit([simple_preprocess(doc) for doc in x])
        return self


data = pd.read_csv('https://pastebin.com/raw/dqKFZ12m')
X_train, X_test, y_train, y_test = train_test_split(data.text, data.label, random_state=0)

classifiers = [LogisticRegression(random_state=0)]

for classifier in classifiers:

    pipeline = Pipeline([
            ('ftt', FTTransformer2(size=10, min_count=0, seed=0)),
            ('clf', classifier)
        ])

    score = cross_val_score(pipeline, X_train, y_train, scoring='accuracy', cv=5)
    print(score)

FastText：无法获取 cross_validation

FastText: Can't get cross_validation

scikit-learn

cross-validation

gensim