按时差阈值匹配数据帧

Question

我有两个数据帧，想通过时间戳来匹配它们。例如：

A    
    Time                X
0   05-01-2017 09:08    3
1   05-01-2017 09:09    6
2   07-01-2017 09:09    5
3   07-01-2017 09:19    4
4   07-01-2017 09:19    8
5   07-02-2017 09:19    7
6   07-02-2017 09:19    5

B    
    Time                Y
0   06-01-2017 14:45    1
1   04-01-2017 03:31    9
2   07-01-2017 03:31    4
3   07-01-2017 14:57    5
4   09-01-2017 14:57    7

数据太多，无法将 df_A 中的每个项目与 df_B 中的每个项目进行比较。相反，我想找到在受控时间阈值内的每个匹配项，例如 2 天。即：

dT = Time A – Time B
-2 < dT < 2

结果应该是：

C                        
Index A Time A          X   Index B Time B          Y   dT
0   05-01-2017 09:08    3   0   06-01-2017 14:45    1   -1.2
0   05-01-2017 09:08    3   1   04-01-2017 03:31    9   1.2
0   05-01-2017 09:08    3   2   07-01-2017 03:31    4   -1.8
1   05-01-2017 09:09    6   0   06-01-2017 14:45    1   -1.2
1   05-01-2017 09:09    6   1   04-01-2017 03:31    9   1.2
1   05-01-2017 09:09    6   2   07-01-2017 03:31    4   -1.8
2   07-01-2017 09:09    5   0   06-01-2017 14:45    1   0.8
2   07-01-2017 09:09    5   2   07-01-2017 03:31    4   0.2
2   07-01-2017 09:09    5   3   07-01-2017 14:57    5   -0.2
3   07-01-2017 09:19    4   0   06-01-2017 14:45    1   0.8
3   07-01-2017 09:19    4   2   07-01-2017 03:31    4   0.2
3   07-01-2017 09:19    4   3   07-01-2017 14:57    5   -0.2
4   07-01-2017 09:19    8   0   06-01-2017 14:45    1   0.8
4   07-01-2017 09:19    8   2   07-01-2017 03:31    4   0.2
4   07-01-2017 09:19    8   3   07-01-2017 14:57    5   -0.2
5   07-02-2017 09:19    7                
6   07-02-2017 09:19    5                
                            4   09-01-2017 14:57    7

我尝试了以下代码，但它不起作用：

import pandas as pd
import datetime as dt
from   datetime import timedelta

# Data
df_A = pd.DataFrame({'X':[3,6,5,4,8,7,5], 'Time_A': [dt.datetime(2017,1,5,9,8),   dt.datetime(2017,1,5,9,9),  dt.datetime(2017,1,7,9,19), dt.datetime(2017,1,7,9,19),  dt.datetime(2017,1,7,9,19), dt.datetime(2017,2,7,9,19), dt.datetime(2017,2,7,9,19)]})
df_B = pd.DataFrame({'Y':[1,9,4,5,7],     'Time_B': [dt.datetime(2017,1,6,14,45), dt.datetime(2017,1,4,3,31), dt.datetime(2017,1,7,3,31), dt.datetime(2017,1,7,14,57), dt.datetime(2017,1,9,14,57)]})

# Match
def slice_datetime(Time, window):

return (Time + timedelta(hours=window)).strftime('%Y-%m-%d %H:%m')

lst = []
for Time in df_A[['X', 'Time_A']].iterrows():
    tmp = df_B.ix[slice_datetime(Time,-48):slice_datetime(Time,48)] # Define the time threshold (hours)
    if not tmp.empty:
        _match = pd.DataFrame()
        for Time_A, (X, Y, Time_B) in tmp.iterrows():
            lst.append([X, Y, Time_A, Time_B])

df_C = pd.DataFrame(lst, columns = ['X', 'Y', 'Time_A', 'Time_B'])

Answer 1

您可以使用时间边界创建两个新列

df_A["start_date"] = df_A["Time_A"]+datetime.timedelta(days=-2)
df_A["end_date"] = df_A["Time_A"]+datetime.timedelta(days=2)

然后加入条件为

的两个dataframe

(df_B.Time_B >= df_A.start_date)&(df_B.Time_B <= df_A.end_date)

希望对您有所帮助！

Answer 2

这里有一个不用循环的想法：

import pandas as pd
df_A = pd.DataFrame({'X':[3,6,5,4,8,7,5], 
                     'Time_A': [pd.datetime(2017,1,5,9,8),   pd.datetime(2017,1,5,9,9),  
                                pd.datetime(2017,1,7,9,19), pd.datetime(2017,1,7,9,19),  
                                pd.datetime(2017,1,7,9,19), pd.datetime(2017,2,7,9,19), 
                                pd.datetime(2017,2,7,9,19)]})
df_B = pd.DataFrame({'Y':[1,9,4,5,7],     
                     'Time_B': [pd.datetime(2017,1,6,14,45), pd.datetime(2017,1,4,3,31), 
                                pd.datetime(2017,1,7,3,31), pd.datetime(2017,1,7,14,57), 
                                pd.datetime(2017,1,9,14,57)]})

#first reset_index and rename
df_A = df_A.reset_index().rename(columns = {'index':'index_A'})
df_B = df_B.reset_index().rename(columns = {'index':'index_B'})

#then create a list of index_B where time_B is within 2 days for each time_A
time_delta = pd.Timedelta(days=2) #check the documentation for more parameter
df_A['list_B'] = (df_A['Time_A'].apply(lambda time_A: 
                    df_B.index_B[(time_A - time_delta <= df_B['Time_B']) & 
                                 (time_A + time_delta >= df_B['Time_B'])].tolist()))

#now use pd.Series and stack, with reset_index drop and rename 
# for finally merge to achieve your goal 
df_C = (df_A.set_index(['index_A','Time_A','X'])['list_B']
            .apply(pd.Series).stack().astype(int)
            .reset_index().drop('level_3',1).rename(columns={0:'index_B'})
            .merge(df_B).sort_values('index_A'))

# Create the columns dT
df_C['dT'] = ((df_C['Time_A'] - df_C['Time_B']).dt.total_seconds()/(24.*3600.)).round(1)

#add the time from df_A and df_B without corresponding time in the other df
# using append and ~ with isin 
df_C = (df_C.append(df_A[~df_A['Time_A'].isin(df_C['Time_A'])].drop('list_B',1))
    .append(df_B[~df_B['Time_B'].isin(df_C['Time_B'])]).fillna(''))

之后您可能需要对列重新排序，但您应该会得到想要的输出

按时差阈值匹配数据帧

Match Dataframes by Time Difference Threshold

python

match

timedelta

dataframe

threshold