OneHotEncoder gives ValueError : Input contains NaN ; even though my DataFrame doesn't contain any NaN as indicated by df.isna()

Question

我正在处理 titanic 数据集，当我尝试在名为 'Embarked' 的列之一上应用 OneHotEncoding 时，它有 3 个可能的值 'S'、'Q' 和 'C'. 它给了我

ValueError: Input contains NaN

我用2种方法检查了栏目内容。第一个是 value_counts 的 for 循环，第二个是将整个 table 写入 csv:

for col in X.columns:
    print(col)
    print(X[col].value_counts(dropna=False))
X.isna().to_csv("xisna.csv")
print("notna================== :",X.notna().shape)
X.dropna(axis=0,how='any',inplace=True)
print("X.shape " ,X.shape)
return pd.DataFrame(X)

产生了

Embarked
S    518
C    139
Q     55
Name: Embarked, dtype: int64

我检查了 csv 的内容并阅读了 700 多个条目，我没有找到任何 'True'-statement。

在 ("cat",One...) 处阻塞的管道

cat_attribs=["Sex","Embarked"]
special_attribs = {'drop_attribs' : ["Name","Cabin","Ticket","PassengerId"], k : [3]}

full_pipeline = ColumnTransformer([
    ("fill",fill_pipeline,list(strat_train_set)),
    ("emb_cat",OneHotEncoder(),['Sex']),
    ("cat",OneHotEncoder(),['Embarked']),
])

那么我缺少的 NaN 值到底在哪里？

Answer 1

我想通了，ColumnTransformer 将连接转换，而不是将它们传递给行中的下一个转换器。因此，在 fill_pipeline 中完成的任何转换都不会被 OneHotEncoder 注意到，因为它仍在使用未转换的数据集。所以我不得不将一个热编码放入 fill_pipeline 而不是 ColumnTransformer。

full_pipeline = ColumnTransformer([
    ("fill",fill_pipeline,list(strat_train_set)),
    ("emb_cat",OneHotEncoder(),['Sex']),
    ("cat",OneHotEncoder(),['Embarked']),
])

OneHotEncoder gives ValueError : Input contains NaN ; even though my DataFrame doesn't contain any NaN as indicated by df.isna()

OneHotEncoder gives ValueError : Input contains NaN ; even though my DataFrame doesn't contain any NaN as indicated by df.isna()

pandas

dataframe

machine-learning

one-hot-encoding

kaggle