使用 mxnet 和 NDarry 训练 CNN 模型

Question

我正在尝试使用 cv2 将 20 张图像转换为 ndarry，然后使用 ndarry 训练 CNN 模型 (mxnet)。首先，我将具有不同像素的彩色图像转换为灰色和 128 x 128 像素图像。所以在第二个 for 循环结束时，我得到一个 (20, 128, 128) ndarry。我使用 ndarry iter 函数 (mxnet) 迭代 ndarry 以进行训练和评估。当我运行其余代码时，我收到以下错误消息：

Traceback (most recent call last):
  File "mx_n.py", line 43, in <module>
    train_iter = mx.io.NDArrayIter(datan[:ntrain, :], label[:ntrain], batch_size, shuffle=True)
IndexError: too many indices for array

代码如下：

import numpy as np
import cv2
import time
import subprocess
import scipy as sp
import glob, os
import mxnet as mx
import pickle
import random
from pandas import *
from sklearn import preprocessing
from PIL import Image
from resizeimage import resizeimage

f_path = "/Users/phillipkim/fd"

img_name = "IMG_17"

filelist = glob.glob(f_path + "/IMG*.jpg")
length = len(filelist) + 1
label = []
data_label = []

for i in range(1,129):
    data_label.append('pixel' + str(i))

for i in range(1,length):
    img = cv2.imread(os.path.join(f_path,img_name) + str(10 + i) + ".jpg")
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    gray = cv2.resize(gray, (128,128))
    cv2.imwrite(os.path.join(f_path,img_name) + "g" + str(10 + i) + ".jpg",gray)
    if(1 <= i <= 10):
        label.append(1)
    elif(11 <= i <= 20):
        label.append(2)

filelist = glob.glob(f_path + "/*g.jpg")
label = np.array(label)
datan = np.array([preprocessing.MinMaxScaler().fit_transform(np.array(Image.open(fname))) for fname in filelist])

batch_size = 4
ntrain = int(datan.shape[0]*0.8)
train_iter = mx.io.NDArrayIter(datan[:ntrain, :], label[:ntrain], batch_size, shuffle=True)
val_iter = mx.io.NDArrayIter(datan[ntrain:, :], label[ntrain:], batch_size)

# Set up the symbolic model
#-------------------------------------------------------------------------------

data = mx.symbol.Variable('data')
# 1st convolutional layer
conv_1 = mx.symbol.Convolution(data = data, kernel = (5, 5), num_filter = 20)
tanh_1 = mx.symbol.Activation(data = conv_1, act_type = "tanh")
pool_1 = mx.symbol.Pooling(data = tanh_1, pool_type = "max", kernel = (2, 2), stride = (2, 2))
# 2nd convolutional layer
conv_2 = mx.symbol.Convolution(data = pool_1, kernel = (5, 5), num_filter = 50)
tanh_2 = mx.symbol.Activation(data = conv_2, act_type = "tanh")
pool_2 = mx.symbol.Pooling(data=tanh_2, pool_type = "max", kernel = (2, 2), stride = (2, 2))
# 1st fully connected layer
flatten = mx.symbol.Flatten(data = pool_2)
fc_1 = mx.symbol.FullyConnected(data = flatten, num_hidden = 500)
tanh_3 = mx.symbol.Activation(data = fc_1, act_type = "tanh")
# 2nd fully connected layer
fc_2 = mx.symbol.FullyConnected(data = tanh_3, num_hidden = 40)
# Output. Softmax output since we'd like to get some probabilities.
NN_model = mx.symbol.SoftmaxOutput(data = fc_2)

# Pre-training set up
#-------------------------------------------------------------------------------

# Device used. CPU in my case.
devices = mx.cpu()

# Training
#-------------------------------------------------------------------------------
train_iter.reset()

# Train the model
model = mx.mod.Module(NN_model,
                        data_names = ['data'],
                        label_names = ['softmax_label'],
                        context = devices)

model.fit(train_iter, eval_data=val_iter, num_epoch = 160,
            optimizer_params={'learning_rate':0.01, 'momentum': 0.9},
            eval_metric = mx.metric.Accuracy(),
            epoch_end_callback = mx.callback.log_train_metric(100))

我将我的代码与 mxnet 网站上的示例代码进行了比较，但我似乎找不到问题所在。你能帮帮我吗？

Answer 1

看来问题是由 datan 引起的。您正在从其他数组的列表中创建一个 numpy 数组，但是如果这些子数组的形状不兼容，您将得到一个数组列表，而不是一个多维数组。当你假设它是一个多维数组而它不是时，你就遇到了问题。

import numpy as np

a = np.array([1,2,3,4])
b = np.array([4,5,6])
data1 = np.array([a, b])
print(data1)
### [array([1, 2, 3, 4]) array([4, 5, 6])]

如果 a 和 b 的形状不兼容（在本例中为不同长度的一维数组），结果将是数组列表，而不是多维数组。

print(data1[1,2])
### IndexError: too many indices for array

所以如果我们在两个轴上切片，我们会得到一个错误。相反，我们必须索引以获取列表中的正确数组，然后再次切片。

print(data1[1][2])
### 6

如果数组是形状兼容的，numpy 会将所有内容连接到一个多维数组中，然后您将能够沿两个轴进行索引。

c = np.array([1,2,3])
d = np.array([4,5,6])
data2 = np.array([c, d])
print(data2)
### [[1 2 3]
### [4 5 6]]

print(data2[1,2])
### 6

因此，要解决此问题，要么更改索引方式，要么强制输入数组具有相同的形状（例如通过裁剪）。

使用 mxnet 和 NDarry 训练 CNN 模型

Training a CNN model using mxnet and NDarry

python

opencv3.0

mxnet

convolutional-neural-network