根据布尔值从 Multi-Index DataFrame 切片中删除行

Question

编辑： 根据要求，我提供了一个更接近我正在使用的真实数据的示例。

所以我有一个 table data 看起来像

            value0    value1    value2
run step                              
0   0      0.12573 -0.132105  0.640423
    1       0.1049 -0.535669  0.361595
    2        1.304  0.947081 -0.703735
    3    -1.265421 -0.623274  0.041326
    4    -2.325031 -0.218792 -1.245911
    5    -0.732267 -0.544259   -0.3163
1   0     0.411631  1.042513 -0.128535
    1     1.366463 -0.665195   0.35151
    2      0.90347  0.094012 -0.743499
    3    -0.921725 -0.457726  0.220195
    4    -1.009618 -0.209176 -0.159225
    5     0.540846  0.214659  0.355373

（认为：时间序列的集合）和第二个 table valid_range

    start   stop
run         
0   1   3
1   2   5

对于每个 run 我想删除所有不满足 start≤step≤stop 的行。

我尝试了以下方法（table 最后生成代码）

for idx in valid_range.index:
    slc = data.loc[idx]
    start, stop = valid_range.loc[idx]
    cond = (start <= slc.index) & (slc.index <= stop)
    data.loc[idx] = data.loc[idx][cond]

但是，这会导致：

         value0 value1 value2
run step                     
0   0       NaN    NaN    NaN
    1       NaN    NaN    NaN
    2       NaN    NaN    NaN
    3       NaN    NaN    NaN
    4       NaN    NaN    NaN
    5       NaN    NaN    NaN
1   0       NaN    NaN    NaN
    1       NaN    NaN    NaN
    2       NaN    NaN    NaN
    3       NaN    NaN    NaN
    4       NaN    NaN    NaN
    5       NaN    NaN    NaN

我也试过data.loc[idx].drop(slc[cond].index, inplace=True)但是没有任何效果...

正在为 table

生成代码

import numpy as np
from pandas import DataFrame, MultiIndex, Index
rng = np.random.default_rng(0)

valid_range = DataFrame({"start": [1, 2], "stop":[3, 5]}, index=Index(range(2), name="run"))
midx = MultiIndex(levels=[[],[]], codes=[[],[]], names=["run", "step"])
data = DataFrame(columns=[f"value{k}" for k in range(3)], index=midx)

for run in range(2):
    for step in range(6):
        data.loc[(run, step), :] = rng.normal(size=(3))
)

Answer 1

我会这样继续分组：

(df.groupby(level=0)
   .apply(lambda x: x[x['small']>1])
   .reset_index(level=0, drop=True)    # remove duplicate index
)

给出：

                           big  small
animal animal attribute              
cow    cow    speed       30.0   20.0
              weight     250.0  150.0
falcon falcon speed      320.0  250.0
lama   lama   speed       45.0   30.0
              weight     200.0  100.0

Answer 2

首先，在'run'的基础上合并data和valid range，使用merge的方法

>>> data

            value0     value1    value2
run step                               
0   0      0.12573  -0.132105  0.640423
    1       0.1049  -0.535669  0.361595
    2        1.304   0.947081 -0.703735
    3     -1.26542  -0.623274  0.041326
    4     -2.32503  -0.218792  -1.24591
    5    -0.732267  -0.544259   -0.3163
1   0     0.411631    1.04251 -0.128535
    1      1.36646  -0.665195   0.35151
    2      0.90347  0.0940123 -0.743499
    3    -0.921725  -0.457726  0.220195
    4     -1.00962  -0.209176 -0.159225
    5     0.540846   0.214659  0.355373


>>> valid_range

     start  stop
run             
0        1     3
1        2     5

>>> merged = data.reset_index().merge(valid_range, how='left', on='run')
>>> merged 

    run  step    value0     value1    value2  start  stop
0     0     0   0.12573  -0.132105  0.640423      1     3
1     0     1    0.1049  -0.535669  0.361595      1     3
2     0     2     1.304   0.947081 -0.703735      1     3
3     0     3  -1.26542  -0.623274  0.041326      1     3
4     0     4  -2.32503  -0.218792  -1.24591      1     3
5     0     5 -0.732267  -0.544259   -0.3163      1     3
6     1     0  0.411631    1.04251 -0.128535      2     5
7     1     1   1.36646  -0.665195   0.35151      2     5
8     1     2   0.90347  0.0940123 -0.743499      2     5
9     1     3 -0.921725  -0.457726  0.220195      2     5
10    1     4  -1.00962  -0.209176 -0.159225      2     5
11    1     5  0.540846   0.214659  0.355373      2     5

然后select使用eval满足条件的行。使用布尔数组屏蔽 data

>>> cond = merged.eval('start < step < stop').to_numpy()
>>> data[cond]

            value0    value1    value2
run step                              
0   2        1.304  0.947081 -0.703735
1   3    -0.921725 -0.457726  0.220195
    4     -1.00962 -0.209176 -0.159225

或者，如果您愿意，这里有一个类似的方法，使用 query

res = (
    data.reset_index()
        .merge(valid_range, on='run', how='left')
        .query('start < step < stop')
        .drop(columns=['start','stop'])
        .set_index(['run', 'step'])
)

根据布尔值从 Multi-Index DataFrame 切片中删除行

Drop rows from a slice of Multi-Index DataFrame based on boolean

python

python-3.x

pandas

dataframe

multi-index