Getting a ValueError: numpy cannot reshape array of size 5342252 to size (5342252, 200, 1) while training lstm

Question

我正在训练一个 lstm 文本生成器来重新创建莎士比亚文本作为学习练习。但是，当我运行它时，我的代码出现错误。 ValueError: cannot reshape array of size 5342252 into shape (5342252,200,1) 我将不胜感激任何帮助，因为我仍在学习 lstm 和 rnn 的基础知识。我的 python 代码：

import numpy
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import LSTM
from keras.callbacks import ModelCheckpoint
from keras.utils import np_utils

filename = "shakespeare.txt"
raw_text = open(filename, 'r').read()

chars = sorted(list(set(raw_text)))
char_to_int = {c:i for i, c in enumerate(chars)}

n_chars = len(raw_text)
n_vocab = len(chars)

seq_length = 200
dataX = []
dataY = []
for i in range(0, n_chars - seq_length):
    seq_in = raw_text[i: i+seq_length]
    seq_out = raw_text[i+seq_length]
    dataX.append([char_to_int[char]] for char in seq_in)
    dataY.append(char_to_int[seq_out])
n_patterns = len(dataX)

dataX = numpy.array(dataX)
X = numpy.reshape(dataX, (n_patterns, seq_length, 1))
X = X / float(n_vocab)
y = np_utils.to_categorical(dataY)

lstm = Sequential()
lstm.add(LSTM(256, input_shape=(X.shape[1], X.shape[2])))
lstm.add(Dropout(0.2))
lstm.add(Dense(y.shape[1], activation='softmax'))
lstm.compile(loss='categorical_crossentropy', optimizer='adam')

filepath = "weights-improvement-{epoch:02d}-{loss:.4f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, 
save_best_only=True, mode='min')
callbacks_list = [checkpoint]

lstm.fit(X, y, epochs=20, batch_size=128, callbacks=callbacks_list)

Answer 1

重塑，

X = numpy.reshape(dataX, (n_patterns, seq_length, 1))

尺寸要一致。如果你想要那个形状，5342252 x 200 x 1 = 1,064,505,600 应该是 dataX 中的元素数。

不清楚你想要完成什么，但我的猜测是

n_patterns = len(dataX)

应该是

n_patterns = len(dataX)/seq_length

Answer 2

这里的问题是 dataX.append(...) 在一个长序列中添加到列表的末尾。你想要做的是构建一个二维数据数组，为此，一个选择是将你的 dataX 和 dataY 声明为 numpy 数组，以开始并附加更多形状的 numpy 数组 (1, seq_length)。请参阅下面的实现

seq_length = 200
dataX = numpy.empty((0,seq_length))
dataY = numpy.empty((0,1))
for i in range(0, n_chars - seq_length):
    seq_in = raw_text[i: i+seq_length]
    seq_out = raw_text[i+seq_length]
    new x = numpy.array([[char_to_int[char]] for char in seq_in]).reshape(1,-1)
    newy = numpy.array([char_to_int[seq_out]]).reshape(1,-1)
    dataX = numpy.append(dataX, newx, axis=0)
    dataY = numpy.append(dataY, newy, axis=0)
n_patterns = len(dataX)

然而，这依赖于扩展数组 dataX 和 dataY，这很慢。更快的方法是预先分配这两个内存：

seq_length = 200
dataX = numpy.zeros((n_chars-seq_length,seq_length))
dataY = numpy.zeros((n_chars-seq_length,1))
for i in range(0, n_chars - seq_length):
    seq_in = raw_text[i: i+seq_length]
    seq_out = raw_text[i+seq_length]
    dataX[i] = [char_to_int[char] for char in seq_in]
    dataY[i] = [char_to_int[seq_out]]
n_patterns = len(dataX)

或者，作为一个非 numpy 选项，您可以在要附加的项目中添加一些方括号

seq_length = 200
dataX = []
dataY = []
print(numpy.shape(dataX))
for i in range(0, n_chars - seq_length):
    seq_in = raw_text[i: i+seq_length]
    seq_out = raw_text[i+seq_length]
    dataX.append([[char_to_int[char]] for char in seq_in])
    dataY.append([char_to_int[seq_out]])
n_patterns = len(dataX)

虽然这会比以前的方法慢。

Getting a ValueError: numpy cannot reshape array of size 5342252 to size (5342252, 200, 1) while training lstm

Getting a ValueError: numpy cannot reshape array of size 5342252 to size (5342252, 200, 1) while training lstm

python

neural-network

lstm

keras

rnn