数据集在循环变量列表时抛出 KeyError
Dataset throwing KeyError while looping through a list of variables
我有一个变量列表，其值的编码方式会导致 Pandas 关闭。例如：我有一个名为 "Alley" 的列，它有一个值列表，其中一个是 NA，它代表 "No Alley"。但是，Pandas 将其解释为 NaN。为了解决这个问题，我使用任意符号（如 XX）对所有 NaN 值进行编码。这些变量实际上没有 null/missing 值。这些只是其值被 Pandas 误解的变量。我将它们收集在一个列表中：
na_data = ['Alley', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'FireplaceQu',
'GarageType', 'GarageFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'MiscFeature']
并将每个 NaN 读数替换为 XX:
for i in na_data:
df[i] = df[i].fillna('XX')
这是我遇到的旧错误：
Traceback (most recent call last):
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 2657, in get_loc
return self._engine.get_loc(key)
File "pandas\_libs\index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index.pyx", line 129, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index_class_helper.pxi", line 91, in pandas._libs.index.Int64Engine._check_type
KeyError: 'Alley'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/security/Downloads/AP/Boston-Kaggle/Model.py", line 67, in <module>
print(feature_encoding(train, categorical_columns))
File "C:/Users/security/Downloads/AP/Boston-Kaggle/Model.py", line 50, in feature_encoding
df[i] = df[i].fillna('XX')
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\pandas\core\frame.py", line 2927, in __getitem__
indexer = self.columns.get_loc(key)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 2659, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas\_libs\index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index.pyx", line 129, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index_class_helper.pxi", line 91, in pandas._libs.index.Int64Engine._check_type
KeyError: 'Alley'
变量Alley肯定存在于数据集中！我 copy/pasta 来自数据集的名称只是为了更好地衡量。
这是我的全部代码（已更新）：
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")
categorical_columns = ['MSSubClass', 'MSZoning', 'LotShape', 'LandContour', 'LotConfig', 'Neighborhood', 'Condition1',
'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd',
'Foundation', 'Heating', 'Electrical', 'Functional', 'GarageType', 'PavedDrive', 'Fence',
'MiscFeature', 'SaleType', 'SaleCondition', 'Street', 'CentralAir', 'Utilities', 'ExterQual',
'LandSlope', 'ExterCond', 'HeatingQC', 'KitchenQual']
ranked_columns = ['Utilities', 'LandSlope', 'ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond', 'BsmtExposure',
'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', 'KitchenQual', 'FireplaceQu', 'GarageQual', 'GarageCond',
'PoolQC', 'OverallQual', 'OverallCond']
numerical_columns = ['LotArea', 'LotFrontage', 'YearBuilt', 'YearRemodAdd', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2',
'BsmtUnfSF','TotalBsmtSF', '1stFlrSF', '2ndFlrSf', 'LowQualFinSF', 'GrLivArea', 'BsmtFullBath',
'BsmtHalfBath', 'FullBath', 'HalfBath', 'Bedroom', 'Kitchen', 'TotRmsAbvGrd', 'Fireplaces',
'GarageYrBlt', 'GarageCars', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch',
'3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal', 'MoSold', 'YrSold']
na_data = ['Alley', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'FireplaceQu',
'GarageType', 'GarageFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'MiscFeature']
for i in na_data:
train[i] = train[i].fillna('XX')
#Replaced the NaN values of LotFrontage and MasVnrArea with the mean of their column
train['LotFrontage'] = train['LotFrontage'].fillna(train['LotFrontage'].mean())
train['MasVnrArea'] = train['MasVnrArea'].fillna(train['MasVnrArea'].mean())
concatenated_list = categorical_columns + na_data
# take one-hot encoding
OHE_sdf = pd.get_dummies(train[concatenated_list])
# drop the old categorical column from original df
train.drop(columns = categorical_columns, axis = 1, inplace = True)
# attach one-hot encoded columns to original data frame
train = pd.concat([train, OHE_sdf], axis = 1, ignore_index = False)
x_train, x_test, y_train, y_test = train_test_split(train, train['SalePrice'], test_size = 0.3, random_state = 42)
sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = 300 * "mean")
sel.fit(x_train, y_train)
sel.get_support()
selected_feat = x_train.columns[sel.get_support()]
print(selected_feat())
这是新错误：
Traceback (most recent call last):
File "/home/onur/Documents/Boston-Kaggle/Model.py", line 49, in <module>
sel.fit(x_train, y_train)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/feature_selection/from_model.py", line 196, in fit
self.estimator_.fit(X, y, **fit_params)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/ensemble/forest.py", line 249, in fit
X = check_array(X, accept_sparse="csc", dtype=DTYPE)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/utils/validation.py", line 496, in check_array
array = np.asarray(array, dtype=dtype, order=order)
File "/opt/anaconda/envs/lib/python3.7/site-packages/numpy/core/_asarray.py", line 85, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: could not convert string to float: 'XX'
你的代码适合我。
import pandas as pd
import numpy as np
df = pd.DataFrame({'x1':[np.nan,2,3,4,5],'x2':[6,7,np.nan,9,10], 'x3':range(10,15)})
list = ['x1', 'x2']
for i in list:
df[i] = df[i].fillna('XX')
您将数据集中在错误的轴上
df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)
# Should be
df = pd.concat([df, OHE_sdf], axis = 0, ignore_index = True)
然而，这将导致另一个错误，即您对 na_columns 中列出的一些列进行了热编码，例如 Garage_Type 已被编码为多个列，每个列对应一个潜在值它不再存在，因此无法替换它的 nan 值。
编辑：
我已经更新了问题代码的几个部分以确保它可以完整运行。
首先我们需要导入我们将要使用的所有库，注意添加了 numpy
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
import numpy as np
其次我们需要从源头获取数据
train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")
现在我们将从数据集中删除所有 NaN
# Create a series of how many NaN's are in each column
nanCounts = train.isna().sum()
# Find the total number of NaN's and print it (used to check that this bits doing somethin)
nanTotal = train.isna().sum().sum()
print('NaN\'s found: ', nanTotal)
# Create a template list
nanCols = []
# Iterate over the series and if the value is more than 0 (i.e there are some NaN's present)
for i in range(0,len(nanCounts)):
if nanCounts[i] > 0:
# If it is append the current column to the list of columns that contain NaN's
nanCols.append(train.columns[i])
# Iterate through all the columns which are known to have NaN's
for i in nanCols:
if train[nanCols][i].dtypes == 'float64':
# If the column is of the data type float64 (a floating point number), replace it with the mean of the column
train[i] = train[i].fillna(train[i].mean())
elif train[nanCols][i].dtypes == 'object':
# If it's of the data type object (a text string) replace it with XX
train[i] = train[i].fillna('XX')
# Reprint the total number of NaN's
nanTotal = train.isna().sum().sum()
print('NaN\'s after removal: ', nanTotal)
现在数据集中没有 NaN，可以 assemble 分类数据列表
# Create a template list
categorical = []
# Iterate across all the columns checking if they're of the object datatype and if they are appending them to the categorical list
for i in range(0, len(train.dtypes)):
if train.dtypes[i] == 'object':
categorical.append(train.columns[i])
# Print out the list of categorical features
print('Categorical columns are: \n', categorical)
现在的代码与原来的代码非常相似，由于变量的变化有一些小的变化
# take one-hot encoding
OHE_sdf = pd.get_dummies(train[categorical])
# drop the old categorical column from original df
train.drop(columns = categorical, axis = 1, inplace = True)
# attach one-hot encoded columns to original data frame
train = pd.concat([train, OHE_sdf], axis = 1, ignore_index = False)
print('splitting dataset')
x_train, x_test, y_train, y_test = train_test_split(train, train['SalePrice'], test_size = 0.3, random_state = 42)
print('Selecting features')
# Note that here i changed the threshold so that it would actually show some features to use
sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = '1.25*mean')
sel.fit(x_train, y_train)
# Also just straight up save the boolean array it will be quicker and i prefer the formatting this way
selected = sel.get_support()
# Print the boolean array of selected features
print(selected)
# Print the finally selected features
print(train.columns[selected])
整体看起来像
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
import numpy as np
train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")
nanCounts = train.isna().sum()
nanTotal = train.isna().sum().sum()
print('NaN\'s found: ', nanTotal)
nanCols = []
for i in range(0,len(nanCounts)):
if nanCounts[i] > 0:
nanCols.append(train.columns[i])
for i in nanCols:
if train[nanCols][i].dtypes == 'float64':
train[i] = train[i].fillna(train[i].mean())
elif train[nanCols][i].dtypes == 'object':
train[i] = train[i].fillna('XX')
nanTotal = train.isna().sum().sum()
print('NaN\'s after removal: ', nanTotal)
categorical = []
for i in range(0, len(train.dtypes)):
if train.dtypes[i] == 'object':
categorical.append(train.columns[i])
print('Categorical columns are: \n', categorical)
# take one-hot encoding
OHE_sdf = pd.get_dummies(train[categorical])
# drop the old categorical column from original df
train.drop(columns = categorical, axis = 1, inplace = True)
# attach one-hot encoded columns to original data frame
train = pd.concat([train, OHE_sdf], axis = 1, ignore_index = False)
print('splitting dataset')
x_train, x_test, y_train, y_test = train_test_split(train, train['SalePrice'], test_size = 0.3, random_state = 42)
print('Selecting features')
sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = '1.25*mean')
sel.fit(x_train, y_train)
selected = sel.get_support()
print(selected)
print(train.columns[selected])
我有一个变量列表，其值的编码方式会导致 Pandas 关闭。例如：我有一个名为 "Alley" 的列，它有一个值列表，其中一个是 NA，它代表 "No Alley"。但是，Pandas 将其解释为 NaN。为了解决这个问题，我使用任意符号（如 XX）对所有 NaN 值进行编码。这些变量实际上没有 null/missing 值。这些只是其值被 Pandas 误解的变量。我将它们收集在一个列表中：
na_data = ['Alley', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'FireplaceQu',
'GarageType', 'GarageFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'MiscFeature']
并将每个 NaN 读数替换为 XX:
for i in na_data:
df[i] = df[i].fillna('XX')
这是我遇到的旧错误：
Traceback (most recent call last):
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 2657, in get_loc
return self._engine.get_loc(key)
File "pandas\_libs\index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index.pyx", line 129, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index_class_helper.pxi", line 91, in pandas._libs.index.Int64Engine._check_type
KeyError: 'Alley'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/security/Downloads/AP/Boston-Kaggle/Model.py", line 67, in <module>
print(feature_encoding(train, categorical_columns))
File "C:/Users/security/Downloads/AP/Boston-Kaggle/Model.py", line 50, in feature_encoding
df[i] = df[i].fillna('XX')
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\pandas\core\frame.py", line 2927, in __getitem__
indexer = self.columns.get_loc(key)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 2659, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas\_libs\index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index.pyx", line 129, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index_class_helper.pxi", line 91, in pandas._libs.index.Int64Engine._check_type
KeyError: 'Alley'
变量Alley肯定存在于数据集中！我 copy/pasta 来自数据集的名称只是为了更好地衡量。
这是我的全部代码（已更新）：
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")
categorical_columns = ['MSSubClass', 'MSZoning', 'LotShape', 'LandContour', 'LotConfig', 'Neighborhood', 'Condition1',
'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd',
'Foundation', 'Heating', 'Electrical', 'Functional', 'GarageType', 'PavedDrive', 'Fence',
'MiscFeature', 'SaleType', 'SaleCondition', 'Street', 'CentralAir', 'Utilities', 'ExterQual',
'LandSlope', 'ExterCond', 'HeatingQC', 'KitchenQual']
ranked_columns = ['Utilities', 'LandSlope', 'ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond', 'BsmtExposure',
'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', 'KitchenQual', 'FireplaceQu', 'GarageQual', 'GarageCond',
'PoolQC', 'OverallQual', 'OverallCond']
numerical_columns = ['LotArea', 'LotFrontage', 'YearBuilt', 'YearRemodAdd', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2',
'BsmtUnfSF','TotalBsmtSF', '1stFlrSF', '2ndFlrSf', 'LowQualFinSF', 'GrLivArea', 'BsmtFullBath',
'BsmtHalfBath', 'FullBath', 'HalfBath', 'Bedroom', 'Kitchen', 'TotRmsAbvGrd', 'Fireplaces',
'GarageYrBlt', 'GarageCars', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch',
'3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal', 'MoSold', 'YrSold']
na_data = ['Alley', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'FireplaceQu',
'GarageType', 'GarageFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'MiscFeature']
for i in na_data:
train[i] = train[i].fillna('XX')
#Replaced the NaN values of LotFrontage and MasVnrArea with the mean of their column
train['LotFrontage'] = train['LotFrontage'].fillna(train['LotFrontage'].mean())
train['MasVnrArea'] = train['MasVnrArea'].fillna(train['MasVnrArea'].mean())
concatenated_list = categorical_columns + na_data
# take one-hot encoding
OHE_sdf = pd.get_dummies(train[concatenated_list])
# drop the old categorical column from original df
train.drop(columns = categorical_columns, axis = 1, inplace = True)
# attach one-hot encoded columns to original data frame
train = pd.concat([train, OHE_sdf], axis = 1, ignore_index = False)
x_train, x_test, y_train, y_test = train_test_split(train, train['SalePrice'], test_size = 0.3, random_state = 42)
sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = 300 * "mean")
sel.fit(x_train, y_train)
sel.get_support()
selected_feat = x_train.columns[sel.get_support()]
print(selected_feat())
这是新错误：
Traceback (most recent call last):
File "/home/onur/Documents/Boston-Kaggle/Model.py", line 49, in <module>
sel.fit(x_train, y_train)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/feature_selection/from_model.py", line 196, in fit
self.estimator_.fit(X, y, **fit_params)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/ensemble/forest.py", line 249, in fit
X = check_array(X, accept_sparse="csc", dtype=DTYPE)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/utils/validation.py", line 496, in check_array
array = np.asarray(array, dtype=dtype, order=order)
File "/opt/anaconda/envs/lib/python3.7/site-packages/numpy/core/_asarray.py", line 85, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: could not convert string to float: 'XX'
你的代码适合我。
import pandas as pd
import numpy as np
df = pd.DataFrame({'x1':[np.nan,2,3,4,5],'x2':[6,7,np.nan,9,10], 'x3':range(10,15)})
list = ['x1', 'x2']
for i in list:
df[i] = df[i].fillna('XX')
您将数据集中在错误的轴上
df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)
# Should be
df = pd.concat([df, OHE_sdf], axis = 0, ignore_index = True)
然而，这将导致另一个错误，即您对 na_columns 中列出的一些列进行了热编码，例如 Garage_Type 已被编码为多个列，每个列对应一个潜在值它不再存在，因此无法替换它的 nan 值。
编辑：
我已经更新了问题代码的几个部分以确保它可以完整运行。
首先我们需要导入我们将要使用的所有库，注意添加了 numpy
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
import numpy as np
其次我们需要从源头获取数据
train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")
现在我们将从数据集中删除所有 NaN
# Create a series of how many NaN's are in each column
nanCounts = train.isna().sum()
# Find the total number of NaN's and print it (used to check that this bits doing somethin)
nanTotal = train.isna().sum().sum()
print('NaN\'s found: ', nanTotal)
# Create a template list
nanCols = []
# Iterate over the series and if the value is more than 0 (i.e there are some NaN's present)
for i in range(0,len(nanCounts)):
if nanCounts[i] > 0:
# If it is append the current column to the list of columns that contain NaN's
nanCols.append(train.columns[i])
# Iterate through all the columns which are known to have NaN's
for i in nanCols:
if train[nanCols][i].dtypes == 'float64':
# If the column is of the data type float64 (a floating point number), replace it with the mean of the column
train[i] = train[i].fillna(train[i].mean())
elif train[nanCols][i].dtypes == 'object':
# If it's of the data type object (a text string) replace it with XX
train[i] = train[i].fillna('XX')
# Reprint the total number of NaN's
nanTotal = train.isna().sum().sum()
print('NaN\'s after removal: ', nanTotal)
现在数据集中没有 NaN，可以 assemble 分类数据列表
# Create a template list
categorical = []
# Iterate across all the columns checking if they're of the object datatype and if they are appending them to the categorical list
for i in range(0, len(train.dtypes)):
if train.dtypes[i] == 'object':
categorical.append(train.columns[i])
# Print out the list of categorical features
print('Categorical columns are: \n', categorical)
现在的代码与原来的代码非常相似，由于变量的变化有一些小的变化
# take one-hot encoding
OHE_sdf = pd.get_dummies(train[categorical])
# drop the old categorical column from original df
train.drop(columns = categorical, axis = 1, inplace = True)
# attach one-hot encoded columns to original data frame
train = pd.concat([train, OHE_sdf], axis = 1, ignore_index = False)
print('splitting dataset')
x_train, x_test, y_train, y_test = train_test_split(train, train['SalePrice'], test_size = 0.3, random_state = 42)
print('Selecting features')
# Note that here i changed the threshold so that it would actually show some features to use
sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = '1.25*mean')
sel.fit(x_train, y_train)
# Also just straight up save the boolean array it will be quicker and i prefer the formatting this way
selected = sel.get_support()
# Print the boolean array of selected features
print(selected)
# Print the finally selected features
print(train.columns[selected])
整体看起来像
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
import numpy as np
train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")
nanCounts = train.isna().sum()
nanTotal = train.isna().sum().sum()
print('NaN\'s found: ', nanTotal)
nanCols = []
for i in range(0,len(nanCounts)):
if nanCounts[i] > 0:
nanCols.append(train.columns[i])
for i in nanCols:
if train[nanCols][i].dtypes == 'float64':
train[i] = train[i].fillna(train[i].mean())
elif train[nanCols][i].dtypes == 'object':
train[i] = train[i].fillna('XX')
nanTotal = train.isna().sum().sum()
print('NaN\'s after removal: ', nanTotal)
categorical = []
for i in range(0, len(train.dtypes)):
if train.dtypes[i] == 'object':
categorical.append(train.columns[i])
print('Categorical columns are: \n', categorical)
# take one-hot encoding
OHE_sdf = pd.get_dummies(train[categorical])
# drop the old categorical column from original df
train.drop(columns = categorical, axis = 1, inplace = True)
# attach one-hot encoded columns to original data frame
train = pd.concat([train, OHE_sdf], axis = 1, ignore_index = False)
print('splitting dataset')
x_train, x_test, y_train, y_test = train_test_split(train, train['SalePrice'], test_size = 0.3, random_state = 42)
print('Selecting features')
sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = '1.25*mean')
sel.fit(x_train, y_train)
selected = sel.get_support()
print(selected)
print(train.columns[selected])