从数值特征中去除异常值

Question

嗨，我正在尝试从具有数字特征的列中删除异常值，但是当我执行我的代码时，整个数据集都被删除了，请问任何人都可以告诉我我做错了什么吗

numerical_columns = data.select_dtypes(include=['int64','float64']).columns.tolist()

print('Number of rows before discarding outlier = %d' % (data.shape[0]))

for i in numerical_columns:

q1 = data[i].quantile(0.25)
q3 = data[i].quantile(0.75)
iqr = q3-q1 #Interquartile range
fence_low  = q1-1.5*iqr
fence_high = q3+1.5*iqr
data = data.loc[(data[i] > fence_low) & (data[i] < fence_high)]

print('Number of rows after discarding outlier = %d' % (data.shape[0]))

Answer 1

下面的代码对我有用。这里 col 是数据框的数字列，您需要为其删除异常值

    #Remove Outliers: keep only the ones that are within +3 to -3 
    # standard deviations in the column   
     df = df[np.abs(df[col]-df[col].mean()) <= (3*df[col].std())]

从数值特征中去除异常值

removing outliers from numerical features

python

data-mining

outliers

data-preprocessing