用 pandas 中的中位数替换离群值

Question

我想用 pandas 做点什么......

我完成了从数据框中分离离群值，但我不知道如何将离群值年龄设置为中值... 我能得到一些帮助吗？

这是我的代码

users = pd.read_table('user.txt', sep='|', index_col='user_id')
print(users)

         age gender occupation zip_code
user_id                                
1         24      M     artist    85711
2         53      F     artist    94043
3         23      M   educator    32067
4         24      M     artist    43537
5         33      F     artist    15213
...      ...    ...        ...      ...
939       26      F    student    33319
940       32      M   engineer    02215
941       20      M    student    97229
942       48      F   educator    78209
943       22      M    student    77841

[943 rows x 4 columns]

users_age = users.reset_index()[['occupation','age']]
users_age_grouped = users_age.groupby('occupation')
users_age['Lower'] = users_age_grouped['age'].transform(
    lambda x: x.quantile(q=.25) - 1.5 * (x.quantile(q=.75) - x.quantile(q=.25)))
users_age['Median'] = users_age_grouped['age'].transform(lambda x: x.quantile(q=.5))
users_age['Upper'] = users_age_grouped['age'].transform(
    lambda x: x.quantile(q=.75) + 1.5 * (x.quantile(q=.75) - x.quantile(q=.25)))

users_age['Outlier'] = (users_age['Lower'] > users_age['age']) |\
    (users_age['Upper'] < users_age['age'])

users_age_outlier = users_age[users_age['Outlier']]
users_age_wo_outlier = users_age[users_age['Outlier'] == False]

print(users_age_outlier)

    occupation  age  Lower  Median  Upper  Outlier
29     student    7    8.5      22   36.5     True
38     student   41    8.5      22   36.5     True
187    student   42    8.5      22   36.5     True
210     artist   66    3.5      31   63.5     True
417    student   55    8.5      22   36.5     True
422     artist   64    3.5      31   63.5     True
558   engineer   69    6.5      36   66.5     True
564    student   40    8.5      22   36.5     True
766   engineer   70    6.5      36   66.5     True
802   engineer   70    6.5      36   66.5     True
838    student   38    8.5      22   36.5     True
860    student   38    8.5      22   36.5     True
914    student   50    8.5      22   36.5     True
925    student   49    8.5      22   36.5     True

我可以使用什么方法？

Answer 1

我认为你需要Series.between with numpy.where:

mask = users_age['age'].between(users_age['Lower'], users_age['Upper'], inclusive=False)

users_age['age'] = np.where(mask, users_age['Median'], users_age['age'])

用 pandas 中的中位数替换离群值

Replacing Outliers with median in pandas

python

outliers

dataframe

pandas