高效的时间序列滑动window函数

Question

我正在尝试为时间序列创建滑动 window。到目前为止，我有一个我设法开始工作的功能，它允许您获取给定的系列，在几秒钟内设置 window 大小，然后创建滚动样本。我的问题是运行需要很长时间，而且似乎是一种低效的方法。

# ========== create dataset  =========================== #

import pandas as pd
from datetime import timedelta, datetime


timestamp_list = ["2022-02-07 11:38:08.625",
                  "2022-02-07 11:38:09.676", 
                  "2022-02-07 11:38:10.084", 
                  "2022-02-07 11:38:10.10000",  
                  "2022-02-07 11:38:11.2320"]

bid_price_list = [1.14338, 
                  1.14341, 
                  1.14340, 
                  1.1434334, 
                  1.1534334]

df = pd.DataFrame.from_dict(zip(timestamp_list, bid_price_list))
df.columns = ['timestamp','value']

# make date time object
df.timestamp = [datetime.strptime(time_i, "%Y-%m-%d %H:%M:%S.%f") for time_i in df.timestamp]

df.head(3)
timestamp   value   timestamp_to_sec
0   2022-02-07 11:38:08.625 1.14338 2022-02-07 11:38:08
1   2022-02-07 11:38:09.676 1.14341 2022-02-07 11:38:09
2   2022-02-07 11:38:10.084 1.14340 2022-02-07 11:38:10

# ========== create rolling time-series function  ====== #


# get the floor of time (second value)
df["timestamp_to_sec"]  = df["timestamp"].dt.floor('s')

# set rollling window length in seconds
window_dt = pd.Timedelta(seconds=2)

# containers for rolling sample statistics
n_list = []
mean_list = []
std_list =[]

# add dt (window) seconds to the original time which was floored to the second
df["timestamp_to_sec_dt"] = df["timestamp_to_sec"]  + window_dt

# get unique end times
time_unique_endlist = np.unique(df.timestamp_to_sec_dt)

# remove end times that are greater than the last actual time, i.e. max(df["timestamp_to_sec"])
time_unique_endlist = time_unique_endlist[time_unique_endlist <= max(df["timestamp_to_sec"])]

# loop running the sliding window (time_i is the end time of each window)
for time_i in time_unique_endlist:
    
    # start time of each rolling window
    start_time = time_i - window_dt
    
    # sample for each time period of sliding window
    rolling_sample = df[(df.timestamp >= start_time) & (df.timestamp <= time_i)]

    
    # calculate the sample statistics
    n_list.append(len(rolling_sample)) # store n observation count
    mean_list.append(rolling_sample.mean()) # store rolling sample mean
    std_list.append(rolling_sample.std()) # store rolling sample standard deviation
    
    # plot histogram for each sample of the rolling sample
    #plt.hist(rolling_sample.value, bins=10)

# tested and n_list brought back the correct values
>>> n_list
[2,3]

是否有更有效的方法，可以改进我的解释的方法或允许我像这样运行滚动 window 的开源包？我知道 pandas 中有 .rolling() 但这会影响值。我想要一些我可以在不均匀间隔数据上使用的东西，使用时间来定义固定滚动 window。

Answer 1

看来这是最好的表现了，希望对大家有帮助。

# set rollling window length in seconds
window_dt = pd.Timedelta(seconds=2)

# add dt seconds to the original timestep
df["timestamp_to_sec_dt"] = df["timestamp_to_sec"]  + window_dt

# unique end time
time_unique_endlist = np.unique(df.timestamp_to_sec_dt)

# remove end values that are greater than the last actual value, i.e. max(df["timestamp_to_sec"])
time_unique_endlist = time_unique_endlist[time_unique_endlist <= max(df["timestamp_to_sec"])]

# containers for rolling sample statistics
mydic = {}
counter = 0

# loop running the rolling window
for time_i in time_unique_endlist:
    
    start_time = time_i - window_dt
    
    # sample for each time period of sliding window
    rolling_sample = df[(df.timestamp >= start_time) & (df.timestamp <= time_i)]

    # calculate the sample statistics
    mydic[counter] = {
                        "sample_size":len(rolling_sample),
                        "sample_mean":rolling_sample["value"].mean(),
                        "sample_std":rolling_sample["value"].std()
                        }
    counter = counter + 1

# results in a DataFrame
results = pd.DataFrame.from_dict(mydic).T

高效的时间序列滑动window函数

Efficient time series sliding window function

python

time-series

dataframe

pandas

rolling-computation