尝试输入大数据帧时 Tensorflow 数据集 API 内存错误

Question

我有 pandas 350K 行和 200 列的数据框。尝试使用数据集 api 构建输入管道时出现内存错误。当我只输入 10K 行时，一切正常，但所有行都没有。此外，当使用 tf.estimator.inputs.pandas_input_fn 时一切正常。

这是代码

x_train, x_test, y_train, y_test = train_test_split(train, labels, test_size=0.25)

feature_columns = [tf.feature_column.numeric_column(c) for c in train.columns
                if train[c].dtype != 'object']

def train_input_fn():
    dataset = tf.data.Dataset.from_tensor_slices((dict(x_train), y_train))
    dataset = dataset.shuffle(1000)
    dataset = dataset.batch(100)
    iterator = dataset.make_one_shot_iterator()
    return iterator.get_next()

model = tf.estimator.DNNClassifier(feature_columns=feature_columns, hidden_units=[20, 2]
model.train(input_fn=train_input_fn, steps=1000)

以及错误信息

INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2018-09-28 14:42:03.736495: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-09-28 14:42:04.070692: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties: 
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.797
pciBusID: 0000:01:00.0
totalMemory: 8.00GiB freeMemory: 6.63GiB
2018-09-28 14:42:04.072060: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0
2018-09-28 14:42:05.139979: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-28 14:42:05.140271: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929]      0 
2018-09-28 14:42:05.140461: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0:   N 
2018-09-28 14:42:05.141143: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6401 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
Traceback (most recent call last):
  File "C:/Users/.../test.py", line 150, in <module>
    nn.train(input_fn=train_input_fn, steps=10)
  ...
  File "C:\Users\...\google\protobuf\text_format.py", line 118, in getvalue
    return self._writer.getvalue()
MemoryError

我尝试设置不同的批量大小、网络架构，但错误仍然存在。

Answer 1

请在此处查看我的回答，了解当您有大量输入数据时创建生成器的代码。

尝试输入大数据帧时 Tensorflow 数据集 API 内存错误

Tensorflow Dataset API memory error when trying to input large dataframe

python

tensorflow

tensorflow-datasets

tensorflow-estimator