滚动 window 重新访问 - 添加 window 滚动数量作为参数 - 向前分析
Rolling window REVISITED - Adding window rolling quantity as a parameter- Walk Forward Analysis
我一直在网上搜索可以创建 滚动 windows 的方法，这样我就可以对时间序列执行称为前向分析的交叉验证技术以广义的方式。
但是，我还没有找到任何在 1) window 大小方面具有灵活性的解决方案（几乎所有方法都有这个；例如，pandas rolling or a bit different np.roll ) 和 2) window 滚动数量，理解为我们想要滚动多少索引 window （即还没有找到包含这个的索引）。
在中@coldspeed的帮助下，我一直在努力优化和制作简洁的代码（我无法在那里发表评论，因为我不未达到所需的声誉；希望尽快到达那里！），但我无法合并 window 滚动数量。
我的想法：
我试过 np.roll 和我下面的例子，但没有成功。
我也尝试修改下面的代码乘以 ith 值，但我没有将它放入我想维护的列表理解中。
3。下面的例子对任何 window 尺寸都很好，但是，它只是 "rolls" 领先 window 一步，我希望它可以推广到任何一步。
那么，¿有没有办法在列表理解方法中使用这两个参数？或者，是否有我没有找到的任何其他资源可以使这更容易？ 非常感谢所有帮助。我的示例代码如下：
In [1]: import numpy as np
In [2]: arr = np.random.random((10,3))
In [3]: arr
Out[3]: array([[0.38020065, 0.22656515, 0.25926935],
[0.13446667, 0.04386083, 0.47210474],
[0.4374763 , 0.20024762, 0.50494097],
[0.49770835, 0.16381492, 0.6410294 ],
[0.9711233 , 0.2004874 , 0.71186102],
[0.61729025, 0.72601898, 0.18970222],
[0.99308981, 0.80017134, 0.64955358],
[0.46632326, 0.37341677, 0.49950571],
[0.45753235, 0.55642914, 0.31972887],
[0.4371343 , 0.08905587, 0.74511753]])
In [4]: inSamplePercentage = 0.4
In [5]: outSamplePercentage = 0.3 * inSamplePercentage
In [6]: windowSizeTrain = round(inSamplePercentage * arr.shape[0])
In [7]: windowSizeTest = round(outSamplePercentage * arr.shape[0])
In [8]: windowTrPlusTs = windowSizeTrain + windowSizeTest
In [9]: sliceListX = [arr[i: i + windowTrPlusTs] for i in range(len(arr) - (windowTrPlusTs-1))]
给定 window 长度 5 和 window 卷数 2，我可以这样指定：
Out [15]:
[array([[0.38020065, 0.22656515, 0.25926935],
[0.13446667, 0.04386083, 0.47210474],
[0.4374763 , 0.20024762, 0.50494097],
[0.49770835, 0.16381492, 0.6410294 ],
[0.9711233 , 0.2004874 , 0.71186102]]),
array([[0.4374763 , 0.20024762, 0.50494097],
[0.49770835, 0.16381492, 0.6410294 ],
[0.9711233 , 0.2004874 , 0.71186102],
[0.61729025, 0.72601898, 0.18970222],
[0.99308981, 0.80017134, 0.64955358]]),
array([[0.9711233 , 0.2004874 , 0.71186102],
[0.61729025, 0.72601898, 0.18970222],
[0.99308981, 0.80017134, 0.64955358],
[0.46632326, 0.37341677, 0.49950571],
[0.45753235, 0.55642914, 0.31972887]]),
array([[0.99308981, 0.80017134, 0.64955358],
[0.46632326, 0.37341677, 0.49950571],
[0.45753235, 0.55642914, 0.31972887],
[0.4371343 , 0.08905587, 0.74511753]])]
（这合并了最后一个数组，尽管它的长度小于 5）。
或：
Out [16]:
[array([[0.38020065, 0.22656515, 0.25926935],
[0.13446667, 0.04386083, 0.47210474],
[0.4374763 , 0.20024762, 0.50494097],
[0.49770835, 0.16381492, 0.6410294 ],
[0.9711233 , 0.2004874 , 0.71186102]]),
array([[0.4374763 , 0.20024762, 0.50494097],
[0.49770835, 0.16381492, 0.6410294 ],
[0.9711233 , 0.2004874 , 0.71186102],
[0.61729025, 0.72601898, 0.18970222],
[0.99308981, 0.80017134, 0.64955358]]),
array([[0.9711233 , 0.2004874 , 0.71186102],
[0.61729025, 0.72601898, 0.18970222],
[0.99308981, 0.80017134, 0.64955358],
[0.46632326, 0.37341677, 0.49950571],
[0.45753235, 0.55642914, 0.31972887]])]
（仅长度为 == 5 的数组 -> 但是，这可以通过简单掩码从上面的数组中导出）。
编辑：忘记提及 this also——如果 pandas 滚动对象支持 iter 方法，则可以做一些事情。
IIUC你想要什么，你可以使用np.lib.stride_tricks.as_strided创建windows尺寸和滚动数量的视图如：
#redefine arr to see better what is happening than with random numbers
arr = np.arange(30).reshape((10,3))
#get arr properties
arr_0, arr_1 = arr.shape
arr_is = arr.itemsize #the size of element in arr
#parameter window and rolling
win_size = 5
roll_qty = 2
# use as_stribed by defining the right parameters:
from numpy.lib.stride_tricks import as_strided
print (as_strided( arr,
shape=(int((arr_0 - win_size)/roll_qty+1), win_size,arr_1),
strides=(roll_qty*arr_1*arr_is, arr_1*arr_is, arr_is)))
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14]],
[[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[18, 19, 20]],
[[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23],
[24, 25, 26]]])
另一个window尺寸和轧制数量：
win_size = 4
roll_qty = 3
print( as_strided( arr,
shape=(int((arr_0 - win_size)/roll_qty+1), win_size,arr_1),
strides=(roll_qty*arr_1*arr_is, arr_1*arr_is, arr_is)))
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]],
[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[18, 19, 20]],
[[18, 19, 20],
[21, 22, 23],
[24, 25, 26],
[27, 28, 29]]])
所以，给我两分钱（在@Ben.T 的帮助下），下面是创建前向分析基本工具的代码，以获得查看您的 model/models 将如何以更普遍的方式执行。
非锚定 WFA
def walkForwardAnal(myArr, windowSize, rollQty):
from numpy.lib.stride_tricks import as_strided
ArrRows, ArrCols = myArr.shape
ArrItems = myArr.itemsize
sliceQtyAndShape = (int((ArrRows - windowSize) / rollQty + 1), windowSize, ArrCols)
print('The final view shape is {}'.format(sliceQtyAndShape))
ArrStrides = (rollQty * ArrCols * ArrItems, ArrCols * ArrItems, ArrItems)
print('The final strides are {}'.format(ArrStrides))
sliceList = list(as_strided(myArr, shape=sliceQtyAndShape, strides=ArrStrides, writeable=False))
return sliceList
wSizeTr = 400
wSizeTe = 100
wSizeTot = wSizeTr + wSizeTe
rQty = 200
sliceListX = wf.walkForwardAnal(X, wSizeTot, rQty)
sliceListY = wf.walkForwardAnal(y, wSizeTot, rQty)
for sliceArrX, sliceArrY in zip(sliceListX, sliceListY):
## Consider having to make a .copy() of each array, so that we don't modify the original one.
# XArr = sliceArrX.copy() and hence, changing Xtrain, Xtest = XArr[...]
# YArr = sliceArrY.copy() and hence, changing Ytrain, Ytest = XArr[...]
Xtrain = sliceArrX[:-wSizeTe,:]
Xtest = sliceArrX[-wSizeTe:,:]
Ytrain = sliceArrY[:-wSizeTe,:]
Ytest = sliceArrY[-wSizeTe:,:]
锚定 WFA
timeSeriesCrossVal = TimeSeriesSplit(n_splits=5)
for trainIndex, testIndex in timeSeriesCrossVal.split(X):
## Check if the training and testing quantities make sense. If not, increase or decrease the n_splits parameter.
Xtrain = X[trainIndex]
Xtest = X[testIndex]
Ytrain = y[trainIndex]
Ytest = y[testIndex]
然后，您可以创建以下内容（使用两种方法中的任何一种）并继续建模：
# Fit on training set only - The targets (y) are already encoded in dummy variables, so no need to standarize them.
scaler = StandardScaler()
scaler.fit(Xtrain)
# Apply transform to both the training set and the test set.
trainX = scaler.transform(Xtrain)
testX = scaler.transform(Xtest)
## PCA - Principal Component Analysis #### APPLY PCA TO THE STANDARIZED TRAINING SET! :::: Fit on training set only.
pca = PCA(.95)
pca.fit(trainX)
# Apply transform to both the training set and the test set.
trainX = pca.transform(trainX)
testX = pca.transform(testX)
## Predict and append predictions...
通用window滚动数量的非锚定情况下的一个衬板：
sliceListX = [arr[i: i + wSizeTot] for i in range(0, arr.shape[0] - wSizeTot+1, rQty)]
我一直在网上搜索可以创建 滚动 windows 的方法，这样我就可以对时间序列执行称为前向分析的交叉验证技术以广义的方式。
但是，我还没有找到任何在 1) window 大小方面具有灵活性的解决方案（几乎所有方法都有这个；例如，pandas rolling or a bit different np.roll ) 和 2) window 滚动数量，理解为我们想要滚动多少索引 window （即还没有找到包含这个的索引）。
在
我的想法：
我试过
np.roll和我下面的例子，但没有成功。我也尝试修改下面的代码乘以
ith值，但我没有将它放入我想维护的列表理解中。
3。下面的例子对任何 window 尺寸都很好，但是，它只是 "rolls" 领先 window 一步，我希望它可以推广到任何一步。
那么，¿有没有办法在列表理解方法中使用这两个参数？或者，是否有我没有找到的任何其他资源可以使这更容易？ 非常感谢所有帮助。我的示例代码如下：
In [1]: import numpy as np
In [2]: arr = np.random.random((10,3))
In [3]: arr
Out[3]: array([[0.38020065, 0.22656515, 0.25926935],
[0.13446667, 0.04386083, 0.47210474],
[0.4374763 , 0.20024762, 0.50494097],
[0.49770835, 0.16381492, 0.6410294 ],
[0.9711233 , 0.2004874 , 0.71186102],
[0.61729025, 0.72601898, 0.18970222],
[0.99308981, 0.80017134, 0.64955358],
[0.46632326, 0.37341677, 0.49950571],
[0.45753235, 0.55642914, 0.31972887],
[0.4371343 , 0.08905587, 0.74511753]])
In [4]: inSamplePercentage = 0.4
In [5]: outSamplePercentage = 0.3 * inSamplePercentage
In [6]: windowSizeTrain = round(inSamplePercentage * arr.shape[0])
In [7]: windowSizeTest = round(outSamplePercentage * arr.shape[0])
In [8]: windowTrPlusTs = windowSizeTrain + windowSizeTest
In [9]: sliceListX = [arr[i: i + windowTrPlusTs] for i in range(len(arr) - (windowTrPlusTs-1))]
给定 window 长度 5 和 window 卷数 2，我可以这样指定：
Out [15]:
[array([[0.38020065, 0.22656515, 0.25926935],
[0.13446667, 0.04386083, 0.47210474],
[0.4374763 , 0.20024762, 0.50494097],
[0.49770835, 0.16381492, 0.6410294 ],
[0.9711233 , 0.2004874 , 0.71186102]]),
array([[0.4374763 , 0.20024762, 0.50494097],
[0.49770835, 0.16381492, 0.6410294 ],
[0.9711233 , 0.2004874 , 0.71186102],
[0.61729025, 0.72601898, 0.18970222],
[0.99308981, 0.80017134, 0.64955358]]),
array([[0.9711233 , 0.2004874 , 0.71186102],
[0.61729025, 0.72601898, 0.18970222],
[0.99308981, 0.80017134, 0.64955358],
[0.46632326, 0.37341677, 0.49950571],
[0.45753235, 0.55642914, 0.31972887]]),
array([[0.99308981, 0.80017134, 0.64955358],
[0.46632326, 0.37341677, 0.49950571],
[0.45753235, 0.55642914, 0.31972887],
[0.4371343 , 0.08905587, 0.74511753]])]
（这合并了最后一个数组，尽管它的长度小于 5）。
或：
Out [16]:
[array([[0.38020065, 0.22656515, 0.25926935],
[0.13446667, 0.04386083, 0.47210474],
[0.4374763 , 0.20024762, 0.50494097],
[0.49770835, 0.16381492, 0.6410294 ],
[0.9711233 , 0.2004874 , 0.71186102]]),
array([[0.4374763 , 0.20024762, 0.50494097],
[0.49770835, 0.16381492, 0.6410294 ],
[0.9711233 , 0.2004874 , 0.71186102],
[0.61729025, 0.72601898, 0.18970222],
[0.99308981, 0.80017134, 0.64955358]]),
array([[0.9711233 , 0.2004874 , 0.71186102],
[0.61729025, 0.72601898, 0.18970222],
[0.99308981, 0.80017134, 0.64955358],
[0.46632326, 0.37341677, 0.49950571],
[0.45753235, 0.55642914, 0.31972887]])]
（仅长度为 == 5 的数组 -> 但是，这可以通过简单掩码从上面的数组中导出）。
编辑：忘记提及 this also——如果 pandas 滚动对象支持 iter 方法，则可以做一些事情。
IIUC你想要什么，你可以使用np.lib.stride_tricks.as_strided创建windows尺寸和滚动数量的视图如：
#redefine arr to see better what is happening than with random numbers
arr = np.arange(30).reshape((10,3))
#get arr properties
arr_0, arr_1 = arr.shape
arr_is = arr.itemsize #the size of element in arr
#parameter window and rolling
win_size = 5
roll_qty = 2
# use as_stribed by defining the right parameters:
from numpy.lib.stride_tricks import as_strided
print (as_strided( arr,
shape=(int((arr_0 - win_size)/roll_qty+1), win_size,arr_1),
strides=(roll_qty*arr_1*arr_is, arr_1*arr_is, arr_is)))
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14]],
[[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[18, 19, 20]],
[[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23],
[24, 25, 26]]])
另一个window尺寸和轧制数量：
win_size = 4
roll_qty = 3
print( as_strided( arr,
shape=(int((arr_0 - win_size)/roll_qty+1), win_size,arr_1),
strides=(roll_qty*arr_1*arr_is, arr_1*arr_is, arr_is)))
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]],
[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[18, 19, 20]],
[[18, 19, 20],
[21, 22, 23],
[24, 25, 26],
[27, 28, 29]]])
所以，给我两分钱（在@Ben.T 的帮助下），下面是创建前向分析基本工具的代码，以获得查看您的 model/models 将如何以更普遍的方式执行。
非锚定 WFA
def walkForwardAnal(myArr, windowSize, rollQty):
from numpy.lib.stride_tricks import as_strided
ArrRows, ArrCols = myArr.shape
ArrItems = myArr.itemsize
sliceQtyAndShape = (int((ArrRows - windowSize) / rollQty + 1), windowSize, ArrCols)
print('The final view shape is {}'.format(sliceQtyAndShape))
ArrStrides = (rollQty * ArrCols * ArrItems, ArrCols * ArrItems, ArrItems)
print('The final strides are {}'.format(ArrStrides))
sliceList = list(as_strided(myArr, shape=sliceQtyAndShape, strides=ArrStrides, writeable=False))
return sliceList
wSizeTr = 400
wSizeTe = 100
wSizeTot = wSizeTr + wSizeTe
rQty = 200
sliceListX = wf.walkForwardAnal(X, wSizeTot, rQty)
sliceListY = wf.walkForwardAnal(y, wSizeTot, rQty)
for sliceArrX, sliceArrY in zip(sliceListX, sliceListY):
## Consider having to make a .copy() of each array, so that we don't modify the original one.
# XArr = sliceArrX.copy() and hence, changing Xtrain, Xtest = XArr[...]
# YArr = sliceArrY.copy() and hence, changing Ytrain, Ytest = XArr[...]
Xtrain = sliceArrX[:-wSizeTe,:]
Xtest = sliceArrX[-wSizeTe:,:]
Ytrain = sliceArrY[:-wSizeTe,:]
Ytest = sliceArrY[-wSizeTe:,:]
锚定 WFA
timeSeriesCrossVal = TimeSeriesSplit(n_splits=5)
for trainIndex, testIndex in timeSeriesCrossVal.split(X):
## Check if the training and testing quantities make sense. If not, increase or decrease the n_splits parameter.
Xtrain = X[trainIndex]
Xtest = X[testIndex]
Ytrain = y[trainIndex]
Ytest = y[testIndex]
然后，您可以创建以下内容（使用两种方法中的任何一种）并继续建模：
# Fit on training set only - The targets (y) are already encoded in dummy variables, so no need to standarize them.
scaler = StandardScaler()
scaler.fit(Xtrain)
# Apply transform to both the training set and the test set.
trainX = scaler.transform(Xtrain)
testX = scaler.transform(Xtest)
## PCA - Principal Component Analysis #### APPLY PCA TO THE STANDARIZED TRAINING SET! :::: Fit on training set only.
pca = PCA(.95)
pca.fit(trainX)
# Apply transform to both the training set and the test set.
trainX = pca.transform(trainX)
testX = pca.transform(testX)
## Predict and append predictions...
通用window滚动数量的非锚定情况下的一个衬板：
sliceListX = [arr[i: i + wSizeTot] for i in range(0, arr.shape[0] - wSizeTot+1, rQty)]