Python 使用函数时无法接受输入
Python can't take input while using functions
我正在研究 Kaggle 上的房价问题。在构建我的模型时，我认为在训练数据集和测试集上重用我一直使用的一些代码是有意义的，所以我将执行相互操作的代码放入一个函数定义中。在此函数中，我正在处理缺失值并使用其 return 执行单热编码并将其用于随机森林回归。但是，它抛出以下错误：
Traceback (most recent call last):
File "C:/Users/security/Downloads/AP/Boston-Kaggle/Model.py", line 56, in <module>
sel.fit(x_train, y_train)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\feature_selection\from_model.py", line 196, in fit
self.estimator_.fit(X, y, **fit_params)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\ensemble\forest.py", line 249, in fit
X = check_array(X, accept_sparse="csc", dtype=DTYPE)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\utils\validation.py", line 542, in check_array
allow_nan=force_all_finite == 'allow-nan')
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\utils\validation.py", line 56, in _assert_all_finite
raise ValueError(msg_err.format(type_err, X.dtype))
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').
我在使用相同代码但未将其组织到函数中时没有遇到此问题。 def feature_selection_and_engineering(df) 是有问题的功能。以下是我的全部代码：
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")
def feature_selection_and_engineering(df):
# Creating a series of how many NaN's are in each column
nan_counts = df.isna().sum()
# Creating a template list
nan_columns = []
# Iterating over the series and if the value is more than 0 (i.e there are some NaN's present)
for i in range(0, len(nan_counts)):
if nan_counts[i] > 0:
nan_columns.append(df.columns[i])
# Iterating through all the columns which are known to have NaN's
for i in nan_columns:
if df[nan_columns][i].dtypes == 'float64':
df[i] = df[i].fillna(df[i].mean())
elif df[nan_columns][i].dtypes == 'object':
df[i] = df[i].fillna('XX')
# Creating a template list
categorical_columns = []
# Iterating across all the columns,
# checking if they're of the object datatype and if they are, appending them to the categorical list
for i in range(0, len(df.dtypes)):
if df.dtypes[i] == 'object':
categorical_columns.append(df.columns[i])
return categorical_columns
# take one-hot encoding
OHE_sdf = pd.get_dummies(feature_selection_and_engineering(train))
# drop the old categorical column from original df
train.drop(columns = feature_selection_and_engineering(train), axis = 1, inplace = True)
# attach one-hot encoded columns to original data frame
train = pd.concat([train, OHE_sdf], axis = 1, ignore_index = False)
# Dividing the training dataset into train/test sets with the test size being 20% of the overall dataset.
x_train, x_test, y_train, y_test = train_test_split(train, train['SalePrice'], test_size = 0.2, random_state = 42)
randomForestRegressor = RandomForestRegressor(n_estimators=1000)
# Invoking the Random Forest Classifier with a 1.25x the mean threshold to select correlating features
sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = '1.25*mean')
sel.fit(x_train, y_train)
selected = sel.get_support()
# linearRegression.fit(x_train, y_train)
randomForestRegressor.fit(x_train, y_train)
# Assigning the accuracy of the model to the variable "accuracy"
accuracy = randomForestRegressor.score(x_train, y_train)
# Predicting for the data in the test set
predictions = randomForestRegressor.predict(feature_selection_and_engineering(test))
# Writing the predictions to a new CSV file
submission = pd.DataFrame({'Id': test['PassengerId'], 'SalePrice': predictions})
filename = 'Boston-Submission.csv'
submission.to_csv(filename, index=False)
print(accuracy*100, "%")
新错误:
Traceback (most recent call last):
File "/home/onur/Documents/Boston-Kaggle/Model.py", line 76, in <module>
x_train, encoder = feature_selection_and_engineering(x_train)
File "/home/onur/Documents/Boston-Kaggle/Model.py", line 57, in feature_selection_and_engineering
encoder = train_one_hot_encoder(df, categorical_columns)
File "/home/onur/Documents/Boston-Kaggle/Model.py", line 30, in train_one_hot_encoder
return enc.fit(categorical_df)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 493, in fit
self._fit(X, handle_unknown=self.handle_unknown)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 80, in _fit
X_list, n_samples, n_features = self._check_X(X)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 67, in _check_X
force_all_finite=needs_validation)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/utils/validation.py", line 542, in check_array
allow_nan=force_all_finite == 'allow-nan')
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/utils/validation.py", line 60, in _assert_all_finite
raise ValueError("Input contains NaN")
ValueError: Input contains NaN
重用代码是个好主意，但要注意将代码放入函数时变量的范围会如何变化。
您遇到的错误是因为您输入随机森林的数组中有 NaN 个值。在您的 feature_engineering_and_selection() 函数中，您正在删除 NaN 值，但是 df 永远不会从函数返回，因此模型中使用了原始的、未修改的 df。
我建议将您的 feature_engineering_and_selection() 函数拆分成不同的组件。在这里，我做了一个函数，只删除 NaNs.
# Iterates through the columns and fixes any NaNs
def remove_nan(df):
replace_dict = {}
for col in df.columns:
# If there are any NaN values in this column
if pd.isna(df[col]).any():
# Replace NaN in object columns with 'N/A'
if df[col].dtypes == 'object':
replace_dict[col] = 'N/A'
# Replace NaN in float columns with 0
elif df[col].dtypes == 'float64':
replace_dict[col] = 0
df = df.fillna(replace_dict)
return df
我建议用 0 代替平均值填充 NaN 数值。对于此数据，有 3 个具有 nan 值的数字列：LotFrontage（连接到 属性 的街道英尺）、MasVnrArea（砖石贴面面积）、GarageYrBlt（车库年份建成）。如果没有车库，那么就没有建造车库的年份，所以将年份设为 0 而不是平均年份等是有意义的。
还有一些工作需要使用您设置的 one hot 编码器完成。创建单热编码可能很棘手，因为训练数据和测试数据需要具有相同的列。如果你有如下训练和测试数据
火车
| House Type |
| ---------- |
| Mansion |
| Ranch |
测试
| House Type |
| ---------- |
| Mansion |
| Duplex |
然后如果使用 pd.get_dummies()，训练列将是 [house_type_mansion, house_type_ranch]，测试列将是 [house_type_mansion, house_type_duplex]，这将不起作用。但是，使用 sklearn，您可以将一个热编码器安装到您的训练数据中。转换测试数据集时，它将创建与训练数据集相同的列。 handle_unknown 参数将告诉编码器如何处理测试集中的 duplex，ignore 或 error。
# Fits an sklearn one hot encoder
def train_one_hot_encoder(df, categorical_columns):
# take one-hot encoding of categorical columns
categorical_df = df[categorical_columns]
enc = OneHotEncoder(sparse=False, handle_unknown='ignore')
return enc.fit(categorical_df)
为了合并分类数据和非分类数据，我再次建议制作一个单独的函数
# One hot encodes the given dataframe
def one_hot_encode(df, categorical_columns, encoder):
# Get dataframe with only categorical columns
categorical_df = df[categorical_columns]
# Get one hot encoding
ohe_df = pd.DataFrame(encoder.transform(categorical_df), columns=encoder.get_feature_names())
# Get float columns
float_df = df.drop(categorical_columns, axis=1)
# Return the combined array
return pd.concat([float_df, ohe_df], axis=1)
最后，您的 feature_engineering_and_selection() 函数可以调用所有这些函数。
def feature_selection_and_engineering(df, encoder=None):
df = remove_nan(df)
categorical_columns = get_categorical_columns(df)
# If there is no encoder, train one
if encoder == None:
encoder = train_one_hot_encoder(df, categorical_columns)
# Encode Data
df = one_hot_encode(df, categorical_columns, encoder)
# Return the encoded data AND encoder
return df, encoder
为了制作代码 运行，我必须修复一些问题，我已将整个修改后的脚本包含在此处的要点中 https://gist.github.com/kylelrichards11/6be90d92a7dd6a5cc9a5290dae3ff94e
我正在研究 Kaggle 上的房价问题。在构建我的模型时，我认为在训练数据集和测试集上重用我一直使用的一些代码是有意义的，所以我将执行相互操作的代码放入一个函数定义中。在此函数中，我正在处理缺失值并使用其 return 执行单热编码并将其用于随机森林回归。但是，它抛出以下错误：
Traceback (most recent call last):
File "C:/Users/security/Downloads/AP/Boston-Kaggle/Model.py", line 56, in <module>
sel.fit(x_train, y_train)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\feature_selection\from_model.py", line 196, in fit
self.estimator_.fit(X, y, **fit_params)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\ensemble\forest.py", line 249, in fit
X = check_array(X, accept_sparse="csc", dtype=DTYPE)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\utils\validation.py", line 542, in check_array
allow_nan=force_all_finite == 'allow-nan')
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\utils\validation.py", line 56, in _assert_all_finite
raise ValueError(msg_err.format(type_err, X.dtype))
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').
我在使用相同代码但未将其组织到函数中时没有遇到此问题。 def feature_selection_and_engineering(df) 是有问题的功能。以下是我的全部代码：
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")
def feature_selection_and_engineering(df):
# Creating a series of how many NaN's are in each column
nan_counts = df.isna().sum()
# Creating a template list
nan_columns = []
# Iterating over the series and if the value is more than 0 (i.e there are some NaN's present)
for i in range(0, len(nan_counts)):
if nan_counts[i] > 0:
nan_columns.append(df.columns[i])
# Iterating through all the columns which are known to have NaN's
for i in nan_columns:
if df[nan_columns][i].dtypes == 'float64':
df[i] = df[i].fillna(df[i].mean())
elif df[nan_columns][i].dtypes == 'object':
df[i] = df[i].fillna('XX')
# Creating a template list
categorical_columns = []
# Iterating across all the columns,
# checking if they're of the object datatype and if they are, appending them to the categorical list
for i in range(0, len(df.dtypes)):
if df.dtypes[i] == 'object':
categorical_columns.append(df.columns[i])
return categorical_columns
# take one-hot encoding
OHE_sdf = pd.get_dummies(feature_selection_and_engineering(train))
# drop the old categorical column from original df
train.drop(columns = feature_selection_and_engineering(train), axis = 1, inplace = True)
# attach one-hot encoded columns to original data frame
train = pd.concat([train, OHE_sdf], axis = 1, ignore_index = False)
# Dividing the training dataset into train/test sets with the test size being 20% of the overall dataset.
x_train, x_test, y_train, y_test = train_test_split(train, train['SalePrice'], test_size = 0.2, random_state = 42)
randomForestRegressor = RandomForestRegressor(n_estimators=1000)
# Invoking the Random Forest Classifier with a 1.25x the mean threshold to select correlating features
sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = '1.25*mean')
sel.fit(x_train, y_train)
selected = sel.get_support()
# linearRegression.fit(x_train, y_train)
randomForestRegressor.fit(x_train, y_train)
# Assigning the accuracy of the model to the variable "accuracy"
accuracy = randomForestRegressor.score(x_train, y_train)
# Predicting for the data in the test set
predictions = randomForestRegressor.predict(feature_selection_and_engineering(test))
# Writing the predictions to a new CSV file
submission = pd.DataFrame({'Id': test['PassengerId'], 'SalePrice': predictions})
filename = 'Boston-Submission.csv'
submission.to_csv(filename, index=False)
print(accuracy*100, "%")
新错误:
Traceback (most recent call last):
File "/home/onur/Documents/Boston-Kaggle/Model.py", line 76, in <module>
x_train, encoder = feature_selection_and_engineering(x_train)
File "/home/onur/Documents/Boston-Kaggle/Model.py", line 57, in feature_selection_and_engineering
encoder = train_one_hot_encoder(df, categorical_columns)
File "/home/onur/Documents/Boston-Kaggle/Model.py", line 30, in train_one_hot_encoder
return enc.fit(categorical_df)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 493, in fit
self._fit(X, handle_unknown=self.handle_unknown)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 80, in _fit
X_list, n_samples, n_features = self._check_X(X)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 67, in _check_X
force_all_finite=needs_validation)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/utils/validation.py", line 542, in check_array
allow_nan=force_all_finite == 'allow-nan')
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/utils/validation.py", line 60, in _assert_all_finite
raise ValueError("Input contains NaN")
ValueError: Input contains NaN
重用代码是个好主意，但要注意将代码放入函数时变量的范围会如何变化。
您遇到的错误是因为您输入随机森林的数组中有 NaN 个值。在您的 feature_engineering_and_selection() 函数中，您正在删除 NaN 值，但是 df 永远不会从函数返回，因此模型中使用了原始的、未修改的 df。
我建议将您的 feature_engineering_and_selection() 函数拆分成不同的组件。在这里，我做了一个函数，只删除 NaNs.
# Iterates through the columns and fixes any NaNs
def remove_nan(df):
replace_dict = {}
for col in df.columns:
# If there are any NaN values in this column
if pd.isna(df[col]).any():
# Replace NaN in object columns with 'N/A'
if df[col].dtypes == 'object':
replace_dict[col] = 'N/A'
# Replace NaN in float columns with 0
elif df[col].dtypes == 'float64':
replace_dict[col] = 0
df = df.fillna(replace_dict)
return df
我建议用 0 代替平均值填充 NaN 数值。对于此数据，有 3 个具有 nan 值的数字列：LotFrontage（连接到 属性 的街道英尺）、MasVnrArea（砖石贴面面积）、GarageYrBlt（车库年份建成）。如果没有车库，那么就没有建造车库的年份，所以将年份设为 0 而不是平均年份等是有意义的。
还有一些工作需要使用您设置的 one hot 编码器完成。创建单热编码可能很棘手，因为训练数据和测试数据需要具有相同的列。如果你有如下训练和测试数据
火车
| House Type |
| ---------- |
| Mansion |
| Ranch |
测试
| House Type |
| ---------- |
| Mansion |
| Duplex |
然后如果使用 pd.get_dummies()，训练列将是 [house_type_mansion, house_type_ranch]，测试列将是 [house_type_mansion, house_type_duplex]，这将不起作用。但是，使用 sklearn，您可以将一个热编码器安装到您的训练数据中。转换测试数据集时，它将创建与训练数据集相同的列。 handle_unknown 参数将告诉编码器如何处理测试集中的 duplex，ignore 或 error。
# Fits an sklearn one hot encoder
def train_one_hot_encoder(df, categorical_columns):
# take one-hot encoding of categorical columns
categorical_df = df[categorical_columns]
enc = OneHotEncoder(sparse=False, handle_unknown='ignore')
return enc.fit(categorical_df)
为了合并分类数据和非分类数据，我再次建议制作一个单独的函数
# One hot encodes the given dataframe
def one_hot_encode(df, categorical_columns, encoder):
# Get dataframe with only categorical columns
categorical_df = df[categorical_columns]
# Get one hot encoding
ohe_df = pd.DataFrame(encoder.transform(categorical_df), columns=encoder.get_feature_names())
# Get float columns
float_df = df.drop(categorical_columns, axis=1)
# Return the combined array
return pd.concat([float_df, ohe_df], axis=1)
最后，您的 feature_engineering_and_selection() 函数可以调用所有这些函数。
def feature_selection_and_engineering(df, encoder=None):
df = remove_nan(df)
categorical_columns = get_categorical_columns(df)
# If there is no encoder, train one
if encoder == None:
encoder = train_one_hot_encoder(df, categorical_columns)
# Encode Data
df = one_hot_encode(df, categorical_columns, encoder)
# Return the encoded data AND encoder
return df, encoder
为了制作代码 运行，我必须修复一些问题，我已将整个修改后的脚本包含在此处的要点中 https://gist.github.com/kylelrichards11/6be90d92a7dd6a5cc9a5290dae3ff94e