Pandas 数据帧与时间间隔对齐

Question

我在尝试对齐两个不同的 pandas 数据帧时遇到问题。实际上时间对齐使用：

import pandas as pd
import datetime
import numpy as np
import matplotlib.pyplot as plt


d1 = np.random.random_integers(0,7000,[4000,1])
d2 = np.random.random_integers(0,7000,[2000,1])

dfA = pd.DataFrame(d1)
dfB = pd.DataFrame(d2)

dfA.columns = ['data1']
dfB.columns = ['data2']

dfA['time'] = pd.date_range('1970-01-01 00:01:00', periods=dfA.shape[0], freq='1S')
dfB['time'] = pd.date_range('1970-01-01 00:00:00', periods=dfB.shape[0], freq='1S')

dfA.set_index('time', inplace=True)
dfB.set_index('time', inplace=True)

dfA1 = dfA.between_time('00:00:00', '00:09:00')
dfA2 = dfA.between_time('00:14:00', '00:16:00')

dfB1 = dfB.between_time('00:00:00', '00:12:00')
dfB2 = dfB.between_time('00:15:00', '00:16:00')

df1 = pd.concat([dfA1, dfA2])
df2 = pd.concat([dfB1, dfB2])


df_aligned = df1.join(df2, how='outer').interpolate(method='time').resample('2S').mean().fillna(method='backfill')

print(df_aligned.head())
df_aligned.plot()

plt.plot(df_aligned['data1'].values)
plt.plot(df_aligned['data2'].values)
plt.show()

然而，df1 和 df2 中的两列显示了不同的时间间隔，因此，我在这些间隔内有新的样本。我的任务只是检索实际数据，没有来自间隙的假样本。

有什么建议吗？非常感谢您。

Answer 1

我找到了解决方案：

1) 首先去掉 interpolate() 并在 fillna() 中设置一个 limit=1。这允许 NaN 值的长时间爆发保留在数据间隙中。当然，您可以根据任务使用您的 fillna 方法和自定义限制。

df_align = df1.join(df2, how='outer').resample('2S').mean().fillna(method='backfill', limit=1)

2) 然后，使用 dropna() 删除所有 NaN 值（即时间间隔内的值）

df_align = df_align.dropna()

最终结果：

import pandas as pd
import datetime
import numpy as np
import matplotlib.pyplot as plt


d1 = np.random.random_integers(0,7000,[4000,1])
d2 = np.random.random_integers(0,7000,[2000,1])

dfA = pd.DataFrame(d1)
dfB = pd.DataFrame(d2)

dfA.columns = ['data1']
dfB.columns = ['data2']

dfA['time'] = pd.date_range('1970-01-01 00:01:00', periods=dfA.shape[0], freq='1S')
dfB['time'] = pd.date_range('1970-01-01 00:00:00', periods=dfB.shape[0], freq='1S')

dfA.set_index('time', inplace=True)
dfB.set_index('time', inplace=True)

dfA1 = dfA.between_time('00:00:00', '00:09:00')
dfA2 = dfA.between_time('00:14:00', '00:16:00')

dfB1 = dfB.between_time('00:00:00', '00:12:00')
dfB2 = dfB.between_time('00:15:00', '00:16:00')

df1 = pd.concat([dfA1, dfA2])
df2 = pd.concat([dfB1, dfB2])


df_aligned = df1.join(df2, how='outer').resample('2S').mean().fillna(method='backfill', limit=1)

df_align = df_align.dropna()    

print(df_aligned.head())
df_aligned.plot()

plt.plot(df_aligned['data1'].values)
plt.plot(df_aligned['data2'].values)
plt.show()

Pandas 数据帧与时间间隔对齐

Pandas data frames alignment with time gaps

python

join

dataframe

pandas

gaps-in-data