编码多列

Question

如果一个数据框有两列或多列数字和文本值，以及一列 Label/Target，如果我想应用像 svm 这样的模型，我该如何只使用我更喜欢的列有兴趣？例如

Data                                     Num       Label/Target   No_Sense
What happens here?                       group1               1   Migrate
Customer Management                      group2               0   Change Stage
Life Cycle Stages                        group1               1   Restructure
Drop-down allows to select status type   group3               1   Restructure Status

等等。

我采用的方法是

1.encode "Num" 列：

one_hot = pd.get_dummies(df['Num'])
df = df.drop('Num',axis = 1)
df = df.join(one_hot)

2.encode "Data" 列：

def bag_words(df):
        
    df = basic_preprocessing(df)
    
    count_vectorizer = CountVectorizer()
    count_vectorizer.fit(df['Data'])
    
    list_corpus = df["Data"].tolist()
    list_labels = df["Label/Target"].tolist()
        
    X = count_vectorizer.transform(list_corpus)
        
    return X, list_labels

然后将bag_words应用于数据集

X, y = bag_words(df)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=40)

我在这些步骤中遗漏了什么吗？如何在我的训练数据集中 select 只有 "Data" 和 "Num" 特征？（因为我认为 "No_Sense" 与我的目的无关）

编辑：我试过

def bag_words(df):
            
    df = basic_preprocessing(df)
        
    count_vectorizer = CountVectorizer()
    count_vectorizer.fit(df['Data'])
        
    list_corpus = df["Data"].tolist()+ df["group1"].tolist()+df["group2"].tolist()+df["group3"].tolist() #<----
    list_labels = df["Label/Target"].tolist()
            
    X = count_vectorizer.transform(list_corpus)
            
    return X, list_labels

但我发现了错误：

TypeError: 'int' object is not iterable

Answer 1

希望对您有所帮助：

import pandas as pd
import numpy as np
import re

from sklearn.feature_extraction.text import CountVectorizer

#this part so I can recreate you df from the string you posted
#remove this part !!!!

data="""
Data                        Num     Label/Target   No_Sense
What happens here?         group1         1          Migrate
Customer Management        group2         0          Change Stage
Life Cycle Stages          group1         1          Restructure
Drop-down allows to select status type  group3   1   Restructure Status
"""
df = pd.DataFrame(np.array( [ re.split(r'\s{2,}', line) for line in lines[1:] ] ), 
                columns = lines[0].split())


#what you want starts from here!!!!:
one_hot = pd.get_dummies(df['Num'])
df = df.drop('Num',axis = 1)
df = df.join(one_hot)

#at this point you have 3 new fetures for 'Num' variable

def bag_words(df):

    

    count_vectorizer = CountVectorizer()
    count_vectorizer.fit(df['Data'])
    matrix = count_vectorizer.transform(df['Data'])

    #this dataframe: `encoded_df`has 15 new features, these are the result of fitting 
    #the CountVectorizer to the 'Data' variable
    encoded_df = pd.DataFrame(data=matrix.toarray(), columns=["Data"+str(i) for i in range(matrix.shape[1])])
    
    #adding them to the dataframe
    df.join(encoded_df)
    
    #getting the numpy arrays that you can use in training
    X = df.loc[:, ["Data"+str(i) for i in range(matrix.shape[1])] + ["group1", "group2", "group3"]].to_numpy()
    y = df.loc[:, ["Label/Target"]].to_numpy()

    return X, y

X, y = bag_words(df)

编码多列

Encoding multiple columns

python

encoding

pandas

one-hot-encoding

countvectorizer