pandas 中的组模式插补(处理 NaN 的组模式)
mode imputation by groups in pandas (handling group modes that are NaN)
我有一个包含 NaN 的分类列“WALLSMATERIAL_MODE”,我想通过以下组 ['NAME_EDUCATION_TYPE'、'AGE_GROUP']:
NAME_EDUCATION_TYPE AGE_GROUP WALLSMATERIAL_MODE
20 Secondary / secondary special 45-60 Stone, brick
21 Secondary / secondary special 21-45 NaN
22 Secondary / secondary special 21-45 Panel
23 Secondary / secondary special 60-70 Mixed
24 Secondary / secondary special 21-45 Panel
25 Secondary / secondary special 45-60 Stone, brick
26 Secondary / secondary special 45-60 Wooden
27 Secondary / secondary special 21-45 NaN
28 Higher education 21-45 NaN
29 Higher education 21-45 Panel
再现性代码
df = pd.DataFrame({'NAME_EDUCATION_TYPE': {20: 'Secondary / secondary special',
21: 'Secondary / secondary special',
22: 'Secondary / secondary special',
23: 'Secondary / secondary special',
24: 'Secondary / secondary special',
25: 'Secondary / secondary special',
26: 'Secondary / secondary special',
27: 'Secondary / secondary special',
28: 'Higher education',
29: 'Higher education'},
'AGE_GROUP': {20: '45-60',
21: '21-45',
22: '21-45',
23: '60-70',
24: '21-45',
25: '45-60',
26: '45-60',
27: '21-45',
28: '21-45',
29: '21-45'},
'WALLSMATERIAL_MODE': {20: 'Stone, brick',
21: np.nan,
22: 'Panel',
23: 'Mixed',
24: 'Panel',
25: 'Stone, brick',
26: 'Wooden',
27: np.nan,
28: np.nan,
29: 'Panel'}})
我尝试从这个 改编以下函数,它适用于中位数插补并处理 NaN
的组中位数
输入:
def mode(s):
if pd.isnull(s.mode()):
return df['WALLSMATERIAL_MODE'].mode()
return s.mode()
df['WALLSMATERIAL_MODE'] = df['WALLSMATERIAL_MODE'].groupby([df['NAME_EDUCATION_TYPE'], df['AGE_GROUP']], dropna=False).apply(lambda x: x.fillna(mode(x)))
OUT:调用pd.isnull时出现以下错误
The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()
我不明白,我尝试在所有组模式上应用pd.isnull,但没有出现此错误。请参阅下面的组模式
输入:
df['WALLSMATERIAL_MODE'].groupby([df['NAME_EDUCATION_TYPE'], df['AGE_GROUP']]).agg(pd.Series.mode).to_dict()
输出:
{('Higher education', '60-70'): nan,
('Higher education', '45-60'): nan,
('Higher education', '21-45'): 'Panel',
('Higher education', '0-21'): nan,
('Secondary / secondary special', '60-70'): 'Mixed',
('Secondary / secondary special', '45-60'): 'Stone, brick',
('Secondary / secondary special', '21-45'): 'Panel',
('Secondary / secondary special', '0-21'): nan}
如果哪位大侠能说出错误在哪里,或者有什么有效的方法可以对本栏目进行分组归类,不胜感激!
下面的代码似乎可以使用 try except 来解决问题。我宁愿避免使用 try except 但我想不出更简洁的方法。
def mode_cats(s):
try:
if pd.isnull(s.mode().any()): # check if the mode of the subgroup is NaN or contains NaN
# (mode() may indeed return a list of several modes)
m = app_train_dash['WALLSMATERIAL_MODE'].mode().iloc[0] # returns the mode of the column
else:
m = s.mode().iloc[0] # returns the mode of the subgroup
return m
except IndexError: # mode returns an empty series if the subgroup consists of a single NaN value
# this causes s.mode().iloc[0] to raise an index error
return app_train_dash['WALLSMATERIAL_MODE'].mode().iloc[0]
正如@Ben.T 所指出的,我必须将 .iloc[0]
与 .mode()
一起使用
但是当 .mode().iloc[0]
有一个空数组作为输入时,我得到 IndexError: single positional indexer is out-of-bounds
。
错误回溯:
- mode() 在值为 NaN 的一行的子组上调用。 .mode() returns 单个 NaN
的子组的空数组
- pd.isnull 在传递的空数组上调用,returns 一个空数组
- 在空数组上调用 .iloc[0] 会引发索引错误
我有一个包含 NaN 的分类列“WALLSMATERIAL_MODE”,我想通过以下组 ['NAME_EDUCATION_TYPE'、'AGE_GROUP']:
NAME_EDUCATION_TYPE AGE_GROUP WALLSMATERIAL_MODE
20 Secondary / secondary special 45-60 Stone, brick
21 Secondary / secondary special 21-45 NaN
22 Secondary / secondary special 21-45 Panel
23 Secondary / secondary special 60-70 Mixed
24 Secondary / secondary special 21-45 Panel
25 Secondary / secondary special 45-60 Stone, brick
26 Secondary / secondary special 45-60 Wooden
27 Secondary / secondary special 21-45 NaN
28 Higher education 21-45 NaN
29 Higher education 21-45 Panel
再现性代码
df = pd.DataFrame({'NAME_EDUCATION_TYPE': {20: 'Secondary / secondary special',
21: 'Secondary / secondary special',
22: 'Secondary / secondary special',
23: 'Secondary / secondary special',
24: 'Secondary / secondary special',
25: 'Secondary / secondary special',
26: 'Secondary / secondary special',
27: 'Secondary / secondary special',
28: 'Higher education',
29: 'Higher education'},
'AGE_GROUP': {20: '45-60',
21: '21-45',
22: '21-45',
23: '60-70',
24: '21-45',
25: '45-60',
26: '45-60',
27: '21-45',
28: '21-45',
29: '21-45'},
'WALLSMATERIAL_MODE': {20: 'Stone, brick',
21: np.nan,
22: 'Panel',
23: 'Mixed',
24: 'Panel',
25: 'Stone, brick',
26: 'Wooden',
27: np.nan,
28: np.nan,
29: 'Panel'}})
我尝试从这个
输入:
def mode(s):
if pd.isnull(s.mode()):
return df['WALLSMATERIAL_MODE'].mode()
return s.mode()
df['WALLSMATERIAL_MODE'] = df['WALLSMATERIAL_MODE'].groupby([df['NAME_EDUCATION_TYPE'], df['AGE_GROUP']], dropna=False).apply(lambda x: x.fillna(mode(x)))
OUT:调用pd.isnull时出现以下错误
The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()
我不明白,我尝试在所有组模式上应用pd.isnull,但没有出现此错误。请参阅下面的组模式
输入:
df['WALLSMATERIAL_MODE'].groupby([df['NAME_EDUCATION_TYPE'], df['AGE_GROUP']]).agg(pd.Series.mode).to_dict()
输出:
{('Higher education', '60-70'): nan,
('Higher education', '45-60'): nan,
('Higher education', '21-45'): 'Panel',
('Higher education', '0-21'): nan,
('Secondary / secondary special', '60-70'): 'Mixed',
('Secondary / secondary special', '45-60'): 'Stone, brick',
('Secondary / secondary special', '21-45'): 'Panel',
('Secondary / secondary special', '0-21'): nan}
如果哪位大侠能说出错误在哪里,或者有什么有效的方法可以对本栏目进行分组归类,不胜感激!
下面的代码似乎可以使用 try except 来解决问题。我宁愿避免使用 try except 但我想不出更简洁的方法。
def mode_cats(s):
try:
if pd.isnull(s.mode().any()): # check if the mode of the subgroup is NaN or contains NaN
# (mode() may indeed return a list of several modes)
m = app_train_dash['WALLSMATERIAL_MODE'].mode().iloc[0] # returns the mode of the column
else:
m = s.mode().iloc[0] # returns the mode of the subgroup
return m
except IndexError: # mode returns an empty series if the subgroup consists of a single NaN value
# this causes s.mode().iloc[0] to raise an index error
return app_train_dash['WALLSMATERIAL_MODE'].mode().iloc[0]
正如@Ben.T 所指出的,我必须将 .iloc[0]
与 .mode()
一起使用
但是当 .mode().iloc[0]
有一个空数组作为输入时,我得到 IndexError: single positional indexer is out-of-bounds
。
错误回溯:
- mode() 在值为 NaN 的一行的子组上调用。 .mode() returns 单个 NaN 的子组的空数组
- pd.isnull 在传递的空数组上调用,returns 一个空数组
- 在空数组上调用 .iloc[0] 会引发索引错误