Keras Generator with Tensorflow Dataset API - IndexError: pop from empty list
Keras Generator with Tensorflow Dataset API - IndexError: pop from empty list
我需要开发一个 RNN 模型，并想使用数据生成器来提供 training/evaluation 循环。
首先，我在从 csv 文件中获取数据时使用了这个帮助功能。
RECORD_DEFAULTS_TRAIN = [[0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]]
def decode_csv(line):
parsed_line = tf.decode_csv(line, RECORD_DEFAULTS_TRAIN)
label = parsed_line[-1] # label is the last element of the list
del parsed_line[-1] # delete the last element from the list
del parsed_line[0] # even delete the first element bcz it is assumed NOT to be a feature
features = tf.stack(parsed_line) # Stack features so that you can later vectorize forward prop., etc.
return features, label
这是我的数据生成器函数：
def data_generator(file_path_list, batch_size):
filenames = tf.placeholder(tf.string, shape=[None])
dataset = tf.data.Dataset.from_tensor_slices(filenames)
dataset = dataset.flat_map(lambda filename: tf.data.TextLineDataset(filename).skip(1).map(decode_csv))
dataset = dataset.shuffle(buffer_size=1000)
dataset = dataset.batch(batch_size)
iterator = dataset.make_initializable_iterator()
next_element = iterator.get_next()
with tf.Session() as sess:
while True:
sess.run(iterator.initializer, feed_dict={filenames: file_path_list})
while True:
try:
batch_data, batch_labels = sess.run(next_element)
# Dimension of the data needs to be: (batch_size, length_of_each_sequence, nr_inputs_in_each_timestep)
# Since the last batch in a epoch can have a different size,
# "batch_data.shape[0]" is used instead of batch_size
batch_data = np.reshape(batch_data, (batch_data.shape[0], SEQUENCE_LEN, 1))
except tf.errors.OutOfRangeError:
break
yield (batch_data, batch_labels)
以下是我构建和训练模型的方式：
lstm_model = Sequential()
lstm_model.add(LSTM(5, input_shape=(SEQUENCE_LEN, 1), return_sequences=True))
lstm_model.add(LSTM(5, input_shape=(SEQUENCE_LEN, 1), return_sequences=False))
lstm_model.add(Dense(1))
opt = tf.keras.optimizers.Adam(lr=0.001, decay=0.0009)
lstm_model.compile(loss='mean_absolute_error', optimizer=opt, metrics=['accuracy'])
lstm_model.fit_generator(data_generator(TRAIN_FILE_PATHS, TRAIN_BATCH_SIZE), #generator,
steps_per_epoch=(NR_TRAIN_EXAMPLES // TRAIN_BATCH_SIZE),
epochs=NR_EPOCHS,
verbose=1,
validation_data=data_generator(DEV_FILE_PATHS, TRAIN_BATCH_SIZE),
validation_steps=(NR_DEV_EXAMPLES // DEV_BATCH_SIZE))
下面是我在测试集上评估模型的方式：
lstm_model.evaluate_generator(data_generator(TEST_FILE_PATHS, TEST_BATCH_SIZE),
steps=(NR_TEST_EXAMPLES // TEST_BATCH_SIZE),
verbose=1)
训练和评估后，我在日志中看到以下错误：
IndexError: pop from empty list
这是训练结束后的最后一部分日志：
Epoch 200/200
2/2 [==============================] - 0s 28ms/step - loss: 0.0091 - acc: 0.0120 - val_loss: 0.0128 - val_acc: 0.0000e+00
Exception ignored in: <generator object data_generator at 0x7fa97d9e34c0>
Traceback (most recent call last):
File "<ipython-input-7-2ef5e6514df7>", line 33, in data_generator
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1530, in __exit__
self._default_graph_context_manager.__exit__(exec_type, exec_value, exec_tb)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 5025, in get_controller
context.context().context_switches.pop()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/context.py", line 136, in pop
self.stack.pop()
IndexError: pop from empty list
Exception ignored in: <generator object data_generator at 0x7fa97d9e3678>
Traceback (most recent call last):
File "<ipython-input-7-2ef5e6514df7>", line 33, in data_generator
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1530, in __exit__
self._default_graph_context_manager.__exit__(exec_type, exec_value, exec_tb)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 5025, in get_controller
context.context().context_switches.pop()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/context.py", line 136, in pop
self.stack.pop()
IndexError: pop from empty list
<tensorflow.python.keras.callbacks.History at 0x7fa97d8b4828>
这是我在 运行 evaluate_generator():
之后看到的
2/2 [==============================] - 0s 28ms/step
Exception ignored in: <generator object data_generator at 0x7fa97d732af0>
Traceback (most recent call last):
File "<ipython-input-7-2ef5e6514df7>", line 33, in data_generator
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1530, in __exit__
self._default_graph_context_manager.__exit__(exec_type, exec_value, exec_tb)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 5025, in get_controller
context.context().context_switches.pop()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/context.py", line 136, in pop
self.stack.pop()
IndexError: pop from empty list
[0.008863004390150309, 0.0]
我感到困惑的是，为什么我在上述每种情况下都会看到错误消息 IndexError: pop from empty list？正常吗？还是我做错了什么？
已解决。
我想解释这个问题而不是删除我的 post，这样它也许也可以帮助其他人..
我只举evaluate_generator(...)函数的例子。
这就是我调用函数的方式..
lstm_model.evaluate_generator(data_generator(TEST_FILE_PATHS, TEST_BATCH_SIZE),
steps=(NR_TEST_EXAMPLES // TEST_BATCH_SIZE),
verbose=1)
然后我将其更改如下：
test_data_generator = data_generator(TEST_FILE_PATHS, TEST_BATCH_SIZE)
lstm_model.evaluate_generator(test_data_generator,
steps=(NR_TEST_EXAMPLES // TEST_BATCH_SIZE),
verbose=1)
问题已解决。我在不同的地方看到了这两种用法，即使在网上找到的每个信息都不一定是真实的。我也不清楚为什么在更改上面的代码时解决了这个问题。如果有人知道，我很乐意听到解释。
我需要开发一个 RNN 模型，并想使用数据生成器来提供 training/evaluation 循环。
首先，我在从 csv 文件中获取数据时使用了这个帮助功能。
RECORD_DEFAULTS_TRAIN = [[0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]]
def decode_csv(line):
parsed_line = tf.decode_csv(line, RECORD_DEFAULTS_TRAIN)
label = parsed_line[-1] # label is the last element of the list
del parsed_line[-1] # delete the last element from the list
del parsed_line[0] # even delete the first element bcz it is assumed NOT to be a feature
features = tf.stack(parsed_line) # Stack features so that you can later vectorize forward prop., etc.
return features, label
这是我的数据生成器函数：
def data_generator(file_path_list, batch_size):
filenames = tf.placeholder(tf.string, shape=[None])
dataset = tf.data.Dataset.from_tensor_slices(filenames)
dataset = dataset.flat_map(lambda filename: tf.data.TextLineDataset(filename).skip(1).map(decode_csv))
dataset = dataset.shuffle(buffer_size=1000)
dataset = dataset.batch(batch_size)
iterator = dataset.make_initializable_iterator()
next_element = iterator.get_next()
with tf.Session() as sess:
while True:
sess.run(iterator.initializer, feed_dict={filenames: file_path_list})
while True:
try:
batch_data, batch_labels = sess.run(next_element)
# Dimension of the data needs to be: (batch_size, length_of_each_sequence, nr_inputs_in_each_timestep)
# Since the last batch in a epoch can have a different size,
# "batch_data.shape[0]" is used instead of batch_size
batch_data = np.reshape(batch_data, (batch_data.shape[0], SEQUENCE_LEN, 1))
except tf.errors.OutOfRangeError:
break
yield (batch_data, batch_labels)
以下是我构建和训练模型的方式：
lstm_model = Sequential()
lstm_model.add(LSTM(5, input_shape=(SEQUENCE_LEN, 1), return_sequences=True))
lstm_model.add(LSTM(5, input_shape=(SEQUENCE_LEN, 1), return_sequences=False))
lstm_model.add(Dense(1))
opt = tf.keras.optimizers.Adam(lr=0.001, decay=0.0009)
lstm_model.compile(loss='mean_absolute_error', optimizer=opt, metrics=['accuracy'])
lstm_model.fit_generator(data_generator(TRAIN_FILE_PATHS, TRAIN_BATCH_SIZE), #generator,
steps_per_epoch=(NR_TRAIN_EXAMPLES // TRAIN_BATCH_SIZE),
epochs=NR_EPOCHS,
verbose=1,
validation_data=data_generator(DEV_FILE_PATHS, TRAIN_BATCH_SIZE),
validation_steps=(NR_DEV_EXAMPLES // DEV_BATCH_SIZE))
下面是我在测试集上评估模型的方式：
lstm_model.evaluate_generator(data_generator(TEST_FILE_PATHS, TEST_BATCH_SIZE),
steps=(NR_TEST_EXAMPLES // TEST_BATCH_SIZE),
verbose=1)
训练和评估后，我在日志中看到以下错误：
IndexError: pop from empty list
这是训练结束后的最后一部分日志：
Epoch 200/200
2/2 [==============================] - 0s 28ms/step - loss: 0.0091 - acc: 0.0120 - val_loss: 0.0128 - val_acc: 0.0000e+00
Exception ignored in: <generator object data_generator at 0x7fa97d9e34c0>
Traceback (most recent call last):
File "<ipython-input-7-2ef5e6514df7>", line 33, in data_generator
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1530, in __exit__
self._default_graph_context_manager.__exit__(exec_type, exec_value, exec_tb)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 5025, in get_controller
context.context().context_switches.pop()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/context.py", line 136, in pop
self.stack.pop()
IndexError: pop from empty list
Exception ignored in: <generator object data_generator at 0x7fa97d9e3678>
Traceback (most recent call last):
File "<ipython-input-7-2ef5e6514df7>", line 33, in data_generator
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1530, in __exit__
self._default_graph_context_manager.__exit__(exec_type, exec_value, exec_tb)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 5025, in get_controller
context.context().context_switches.pop()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/context.py", line 136, in pop
self.stack.pop()
IndexError: pop from empty list
<tensorflow.python.keras.callbacks.History at 0x7fa97d8b4828>
这是我在 运行 evaluate_generator():
2/2 [==============================] - 0s 28ms/step
Exception ignored in: <generator object data_generator at 0x7fa97d732af0>
Traceback (most recent call last):
File "<ipython-input-7-2ef5e6514df7>", line 33, in data_generator
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1530, in __exit__
self._default_graph_context_manager.__exit__(exec_type, exec_value, exec_tb)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 5025, in get_controller
context.context().context_switches.pop()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/context.py", line 136, in pop
self.stack.pop()
IndexError: pop from empty list
[0.008863004390150309, 0.0]
我感到困惑的是，为什么我在上述每种情况下都会看到错误消息 IndexError: pop from empty list？正常吗？还是我做错了什么？
已解决。 我想解释这个问题而不是删除我的 post，这样它也许也可以帮助其他人..
我只举evaluate_generator(...)函数的例子。
这就是我调用函数的方式..
lstm_model.evaluate_generator(data_generator(TEST_FILE_PATHS, TEST_BATCH_SIZE),
steps=(NR_TEST_EXAMPLES // TEST_BATCH_SIZE),
verbose=1)
然后我将其更改如下：
test_data_generator = data_generator(TEST_FILE_PATHS, TEST_BATCH_SIZE)
lstm_model.evaluate_generator(test_data_generator,
steps=(NR_TEST_EXAMPLES // TEST_BATCH_SIZE),
verbose=1)
问题已解决。我在不同的地方看到了这两种用法，即使在网上找到的每个信息都不一定是真实的。我也不清楚为什么在更改上面的代码时解决了这个问题。如果有人知道，我很乐意听到解释。