如何将函数应用于 Pandas 中的多个多索引列?

How to apply a function to multiple multiindex columns in Pandas?

给定一个多索引列

         a                      ...                              
         E1                      ...        E3                    
         g1        g2        g3  ...        g1        g2        g3
0  0.548814  0.715189  0.602763  ...  0.437587  0.891773  0.963663
1  0.383442  0.791725  0.528895  ...  0.087129  0.020218  0.832620
2  0.778157  0.870012  0.978618  ...  0.118274  0.639921  0.143353
3  0.944669  0.521848  0.414662  ...  0.568434  0.018790  0.617635
4  0.612096  0.616934  0.943748  ...  0.697631  0.060225  0.666767
5  0.670638  0.210383  0.128926  ...  0.438602  0.988374  0.102045
6  0.208877  0.161310  0.653108  ...  0.158970  0.110375  0.656330
7  0.138183  0.196582  0.368725  ...  0.096098  0.976459  0.468651
8  0.976761  0.604846  0.739264  ...  0.296140  0.118728  0.317983
9  0.414263  0.064147  0.692472  ...  0.093941  0.575946  0.929296
[10 rows x 9 columns]

我想将第二级过滤的多列(即 E1E2E3)应用到一个函数(例如,ration_type1ration_type2,实际执行中还可以更多)。

例如。假设我们要在函数 ration_type1ration_type2 下计算 E1 的第二级。那么我们只处理下面的df

    a                    
         E1                    
         g1        g2        g3
0  0.548814  0.715189  0.602763
1  0.383442  0.791725  0.528895
  .................
8  0.976761  0.604846  0.739264
9  0.414263  0.064147  0.692472

为了概括所有第二级,我依赖于下面的列表理解

对于每个 ration_type1ration_type2

all_df1 = [ration_type1(df.loc[:, (slice(None), dgroup, slice(None))]) for dgroup in [`E1`, `E2`, `E3`]]


all_df2 = [ration_type2(df.loc[:, (slice(None), dgroup, slice(None))]) for dgroup in [`E1`, `E2`, `E3`]]

在将它连接回原来的 df 之前

但是,我想知道是否有比 list comprehension 方法更优雅和紧凑的方法。这是因为,在现实生活中,可以有更多的配给函数。

完整代码如下

 import numpy as np

import pandas as pd

np.random.seed(0)

arr = np.random.rand(10,9)

tuples = [('a', 'E1', 'g1'), ('a', 'E1', 'g2'), ('a', 'E1', 'g3'), ('a', 'E2', 'g1'), ('a', 'E2', 'g2'),
          ('a', 'E2', 'g3'), ('a', 'E3', 'g1'), ('a', 'E3', 'g2'), ('a', 'E3', 'g3')]
df = pd.DataFrame(data=arr, columns=pd.MultiIndex.from_tuples(tuples))

print(df)
def ration_type1(df):
    """
    (g3+g2)/g1
    # Ugly way since have to convert to numpy 1st
    """

    print(df)
    dration = 'ration_type1'
    l1, l2, _ = df.columns.tolist()[0]
    total = df.loc[:, (slice(None), slice(None), 'g2')].to_numpy() + \
            df.loc[:, (slice(None), slice(None), 'g3')].to_numpy()
    arr = total / df.loc[:, (slice(None), slice(None), 'g1')].to_numpy()

    return pd.DataFrame(data=arr, columns=pd.MultiIndex.from_tuples([(l1, l2, dration)]))


def ration_type2(df):
    """
    (g3+g2+g1)/g1
    # Ugly way since have to convert to numpy 1st
    """
    dration = 'ration_type2'
    l1, l2, _ = df.columns.tolist()[0]
    total = df.loc[:, (slice(None), slice(None), 'g1')].to_numpy() + \
            df.loc[:, (slice(None), slice(None), 'g2')].to_numpy() + \
            df.loc[:, (slice(None), slice(None), 'g3')].to_numpy()
    arr = total / df.loc[:, (slice(None), slice(None), 'g1')].to_numpy()

    return pd.DataFrame(data=arr, columns=pd.MultiIndex.from_tuples([(l1, l2, dration)]))


level1_name = list(set(df.columns.get_level_values(1)))

all_df1 = [ration_type1(df.loc[:, (slice(None), dgroup, slice(None))]) for dgroup in level1_name]
all_df2 = [ration_type2(df.loc[:, (slice(None), dgroup, slice(None))]) for dgroup in level1_name]

df1 = pd.concat(all_df1, axis=1)
df2 = pd.concat(all_df2, axis=1)

df=pd.concat([df,df1,df2],axis=1)

预期输出。

          a                      ...                                       
         E1                      ...                        E2           E3
         g1        g2        g3  ... ration_type2 ration_type2 ration_type2
0  0.548814  0.715189  0.602763  ...     3.401458     2.962896     5.240151
1  0.383442  0.791725  0.528895  ...     4.444124     2.754497    10.788191
2  0.778157  0.870012  0.978618  ...     3.375653     2.554145     7.622516
3  0.944669  0.521848  0.414662  ...     1.991363     5.650758     2.119612
4  0.612096  0.616934  0.943748  ...     3.549735     2.168255     2.042087
5  0.670638  0.210383  0.128926  ...     1.505949     3.960760     3.486126
6  0.208877  0.161310  0.653108  ...     4.899035     3.806001     5.822965
7  0.138183  0.196582  0.368725  ...     5.091008     2.138921    16.037821
8  0.976761  0.604846  0.739264  ...     2.376088    11.283905     2.474676
9  0.414263  0.064147  0.692472  ...     2.826423     2.391873    17.023361

[10 rows x 15 columns]

我正在考虑使用 apply

# function for prepending 'Geek'
def multiply_by_2(number):
    return 2 * number
 
# executing the function
df[["Integers", "Float"]] = df[["Integers", "Float"]].apply(multiply_by_2)

但是,由于我的示例涉及多索引列,因此我很难(由于我的知识有限)

如果使用 MultiIndex 则不那么容易 - 将 g 值重命名为 ration_type1, ration_type2 的解决方案过滤器级别可能划分 MultiIndex DataFrames:

idx = pd.IndexSlice
c = {'g1':'ration_type1','g2':'ration_type1','g3':'ration_type1'}
df1 = df.loc[:, idx[:,:,['g3','g2']]].rename(columns=c).groupby(level=[0,1,2], axis=1).sum()
df11 = df1.div(df.xs('g1', level=2, axis=1, drop_level=False).rename(columns=c))


c1 = {'g1':'ration_type2','g2':'ration_type2','g3':'ration_type2'}
df2 = df.rename(columns=c1).groupby(level=[0,1,2], axis=1).sum()
df22 = df2.div(df.xs('g1', level=2, axis=1, drop_level=False).rename(columns=c1))

df=pd.concat([df,df11,df22],axis=1)

最简单的先reshape:

df1 = df.stack([0,1])
df1['ration_type1'] = df1[['g2','g3']].sum(axis=1).div(df1['g1'])
df1['ration_type2'] = df1.sum(axis=1).div(df1['g1'])

print(df1)
              g1        g2        g3  ration_type1  ration_type2
0 a E1  0.548814  0.715189  0.602763      2.401458      7.777186
    E2  0.544883  0.423655  0.645894      1.962896      6.565312
    E3  0.437587  0.891773  0.963663      4.240151     14.929992
1 a E1  0.383442  0.791725  0.528895      3.444124     13.426259
    E2  0.568045  0.925597  0.071036      1.754497      5.843159
    E3  0.087129  0.020218  0.832620      9.788191    123.129174
2 a E1  0.778157  0.870012  0.978618      2.375653      6.428577
    E2  0.799159  0.461479  0.780529      1.554145      4.498872
    E3  0.118274  0.639921  0.143353      6.622516     63.615316
3 a E1  0.944669  0.521848  0.414662      0.991363      3.040793
    E2  0.264556  0.774234  0.456150      4.650758     23.230266
    E3  0.568434  0.018790  0.617635      1.119612      4.089254
4 a E1  0.612096  0.616934  0.943748      2.549735      7.715318
    E2  0.681820  0.359508  0.437032      1.168255      3.881690
    E3  0.697631  0.060225  0.666767      1.042087      3.535837
5 a E1  0.670638  0.210383  0.128926      0.505949      2.260380
    E2  0.315428  0.363711  0.570197      2.960760     13.347233
    E3  0.438602  0.988374  0.102045      2.486126      9.154429
6 a E1  0.208877  0.161310  0.653108      3.899035     23.565714
    E2  0.253292  0.466311  0.244426      2.806001     14.884143
    E3  0.158970  0.110375  0.656330      4.822965     36.161882
7 a E1  0.138183  0.196582  0.368725      4.091008     34.696743
    E2  0.820993  0.097101  0.837945      1.138921      3.526168
    E3  0.096098  0.976459  0.468651     15.037821    172.521382
8 a E1  0.976761  0.604846  0.739264      1.376088      3.784915
    E2  0.039188  0.282807  0.120197     10.283905    273.710140
    E3  0.296140  0.118728  0.317983      1.474676      7.454332
9 a E1  0.414263  0.064147  0.692472      1.826423      7.235273
    E2  0.566601  0.265389  0.523248      1.391873      4.848404
    E3  0.093941  0.575946  0.929296     16.023361    187.592593

最后一次整形为原始 MultiIndex

df = df1.unstack([1,2]).reorder_levels([1,2,0], axis=1)