重采样数据 - 使用 imblearn 中的 SMOTE 和 3D numpy 数组

Question

我想重新采样我的数据集。这包括标签为 3 classes 的分类转换数据。每个 class 的样本量为：

class 的计数 A：6945
class B 的计数：650
计数 class C: 9066
样本总数：16661

没有标签的数据形状是(16661, 1000, 256)。这意味着 (1000,256) 的 16661 个样本。我想要的是将数据上采样到大多数 class 中的样本数，即 class A -> (6945)

然而，调用时：

from imblearn.over_sampling import SMOTE
print(categorical_vector.shape)
sm = SMOTE(random_state=2)
X_train_res, y_labels_res = sm.fit_sample(categorical_vector, labels.ravel())

它一直说 ValueError: 找到暗淡 3 的数组。Estimator expected <= 2。

我怎样才能以估计器可以适合它并且也有意义的方式展平数据？此外，在获得 X_train_res?

后如何展开（使用 3D 维度）

Answer 1

我正在考虑一个虚拟的 3d 数组并自己假设一个 2d 数组大小，

arr = np.random.rand(160, 10, 25)
orig_shape = arr.shape
print(orig_shape)

输出：(160, 10, 25)

arr = np.reshape(arr, (arr.shape[0], arr.shape[1]))
print(arr.shape)

输出：(4000, 10)

arr = np.reshape(arr, orig_shape))
print(arr.shape)

输出：(160, 10, 25)

Answer 2

from imblearn.over_sampling 
import RandomOverSampler 
import numpy as np 
oversample = RandomOverSampler(sampling_strategy='minority')

X 可以是像 X[sample,time,feature] 这样的时间步进 3D 数据，而 y 就像每个样本的二进制值。例如：(1,1),(2,1),(3,1) -> 1

X = np.array([[[1,1],[2,1],[3,1]],
             [[2,1],[3,1],[4,1]],
             [[5,1],[6,1],[7,1]],
             [[8,1],[9,1],[10,1]],
             [[11,1],[12,1],[13,1]]
             ])

y = np.array([1,0,1,1,0])

无法使用 3D X 值训练 OVERSAMPLER，因为如果您使用 2D，您将得到 2D 数据。

Xo,yo = oversample.fit_resample(X[:,:,0], y)
Xo:
[[ 1  2  3]
 [ 2  3  4]
 [ 5  6  7]
 [ 8  9 10]
 [11 12 13]
 [ 2  3  4]]

yo:
[1 0 1 1 0 0]

但是如果你使用2D数据(sample,time,0)来拟合模型，它会返回索引，这足以创建3D过采样数据

oversample.fit_resample(X[:,:,0], y)
Xo = X[oversample.sample_indices_]
yo = y[oversample.sample_indices_]

Xo:
[[[ 1  1][ 2  1][ 3  1]]
 [[ 2  1][ 3  1][ 4  1]]
 [[ 5  1][ 6  1][ 7  1]]
 [[ 8  1][ 9  1][10  1]]
 [[11  1][12  1][13  1]]
 [[ 2  1][ 3  1][ 4  1]]]
yo:
[1 0 1 1 0 0]

Answer 3

我将为 2 维数组创建每个点，然后将其重塑为 3 维数组。我已经提供了我的脚本。如有不解，请评论；请回复

x_train, y_train = zip(*train_dataset)
x_test, y_test = zip(*test_dataset)

dim_1 = np.array(x_train).shape[0]
dim_2 = np.array(x_train).shape[1]
dim_3 = np.array(x_train).shape[2]

new_dim = dim_1 * dim_2

new_x_train = np.array(x_train).reshape(new_dim, dim_3)


new_y_train = []
for i in range(len(y_train)):
    # print(y_train[i])
    new_y_train.extend([y_train[i]]*dim_2)

new_y_train = np.array(new_y_train)

# transform the dataset
oversample = SMOTE()
X_Train, Y_Train = oversample.fit_sample(new_x_train, new_y_train)
# summarize the new class distribution
counter = Counter(Y_Train)
print('The number of samples in TRAIN: ', counter)



x_train_SMOTE = X_Train.reshape(int(X_Train.shape[0]/dim_2), dim_2, dim_3)

y_train_SMOTE = []
for i in range(int(X_Train.shape[0]/dim_2)):
    # print(i)
    value_list = list(Y_Train.reshape(int(X_Train.shape[0]/dim_2), dim_2)[i])
    # print(list(set(value_list)))
    y_train_SMOTE.extend(list(set(value_list)))
    ## Check: if there is any different value in a list 
    if len(set(value_list)) != 1:
        print('\n\n********* STOP: THERE IS SOMETHING WRONG IN TRAIN ******\n\n')
    


dim_1 = np.array(x_test).shape[0]
dim_2 = np.array(x_test).shape[1]
dim_3 = np.array(x_test).shape[2]

new_dim = dim_1 * dim_2

new_x_test = np.array(x_test).reshape(new_dim, dim_3)


new_y_test = []
for i in range(len(y_test)):
    # print(y_train[i])
    new_y_test.extend([y_test[i]]*dim_2)

new_y_test = np.array(new_y_test)

# transform the dataset
oversample = SMOTE()
X_Test, Y_Test = oversample.fit_sample(new_x_test, new_y_test)
# summarize the new class distribution
counter = Counter(Y_Test)
print('The number of samples in TEST: ', counter)



x_test_SMOTE = X_Test.reshape(int(X_Test.shape[0]/dim_2), dim_2, dim_3)

y_test_SMOTE = []
for i in range(int(X_Test.shape[0]/dim_2)):
    # print(i)
    value_list = list(Y_Test.reshape(int(X_Test.shape[0]/dim_2), dim_2)[i])
    # print(list(set(value_list)))
    y_test_SMOTE.extend(list(set(value_list)))
    ## Check: if there is any different value in a list 
    if len(set(value_list)) != 1:
        print('\n\n********* STOP: THERE IS SOMETHING WRONG IN TEST ******\n\n')

重采样数据 - 使用 imblearn 中的 SMOTE 和 3D numpy 数组

resampling data - using SMOTE from imblearn with 3D numpy arrays

python

numpy

imblearn