如何使用 tensorflow 数据集读取多个 .mat 文件（太大而无法放入内存）

Question

我有大约 550K 个样本，每个样本都是 200x50x1。该数据集的大小约为 57GB。

我想在此集合上训练一个网络，但我无法阅读它。

batch_size=8

def _read_py_function(filename,labels_slice):
    with h5py.File(filename, 'r') as f:
        data_slice = np.asarray(f['feats'])
        print(data_slice.shape)
    return data_slice, labels_slice

placeholder_files = tf.placeholder(tf.string, [None])
placeholder_labels = tf.placeholder(tf.int32, [None])

dataset = tf.data.Dataset.from_tensor_slices((placeholder_files,placeholder_labels))
dataset = dataset.map(
    lambda filename, label: tuple(tf.py_func(
        _read_py_function, [filename,label], [tf.uint8, tf.int32])))

dataset = dataset.shuffle(buffer_size=50000)
dataset = dataset.batch(batch_size)

iterator = tf.data.Iterator.from_structure(dataset.output_types, dataset.output_shapes)
data_X, data_y = iterator.get_next()
data_y = tf.cast(data_y, tf.int32)

net = conv_layer(inputs=data_X,num_outputs=8, kernel_size=3, stride=2, scope='rcl_0')
net = pool_layer(inputs=net,kernel_size=2,scope='pl_0')

net = dropout_layer(inputs=net,scope='dl_0')

net = flatten_layer(inputs=net,scope='flatten_0')
net = dense_layer(inputs=net,num_outputs=256,scope='dense_0')
net = dense_layer(inputs=net,num_outputs=64,scope='dense_1')
out = dense_layer(inputs=net,num_outputs=10,scope='dense_2')

我运行会话使用 :

sess.run(train_iterator, feed_dict = {placeholder_files: filenames, placeholder_labels: ytrain})
        try:
            while True:
                _, loss, acc = sess.run([train_op, loss_op, accuracy_op])
                train_loss += loss 
                train_accuracy += acc
        except tf.errors.OutOfRangeError:
            pass

但我什至在运行开启会话之前就收到了错误：

Traceback (most recent call last):
  File "SFCC-trial-134.py", line 297, in <module>
    net = rcnn_layer(inputs=data_X,num_outputs=8, kernel_size=3, stride=2, scope='rcl_0')
  File "SFCC-trial-134.py", line 123, in rcnn_layer
    reuse=False)
  File "SFCC-trial-134.py", line 109, in conv_layer
    reuse         = reuse
  File "/home/priyam.jain/tensorflow-gpu-python3/lib/python3.5/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
    return func(*args, **current_args)
  File "/home/priyam.jain/tensorflow-gpu-python3/lib/python3.5/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1154, in convolution2d
    conv_dims=2)
  File "/home/priyam.jain/tensorflow-gpu-python3/lib/python3.5/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
    return func(*args, **current_args)
  File "/home/priyam.jain/tensorflow-gpu-python3/lib/python3.5/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1025, in convolution
    (conv_dims + 2, input_rank))
TypeError: %d format: a number is required, not NoneType

我虽然考虑过使用 TFRecords，但很难创建它们。在我学习为我的数据集创建它们的地方找不到好的 post。

conv_layer定义如下：

def conv_layer(inputs, num_outputs, kernel_size, stride, normalizer_fn=None, activation_fn=nn.relu, trainable=True, scope='noname', reuse=False):

    net = slim.conv2d(inputs = inputs,
        num_outputs   = num_outputs,
        kernel_size   = kernel_size,
        stride        = stride,
        normalizer_fn = normalizer_fn,
        activation_fn = activation_fn,
        trainable     = trainable,
        scope         = scope,
        reuse         = reuse
        )
    return net

Answer 1

不要在地图函数中传递 tf.py_func。您可以通过直接在 map 函数中传递函数名称来读取文件图像。我只列出了代码的相关部分。

def _read_py_function(filename, label):
    return tf.zeros((224, 224, 3), dtype=tf.float32), tf.ones((1,), dtype=tf.int32)

dataset = dataset.map(lambda filename, label: _read_py_function(filename, label))

另一个变化是您的 iterator 将只需要浮点输入。因此，您必须将 tf.uint8 类型的输出更改为 float.

如何使用 tensorflow 数据集读取多个 .mat 文件（太大而无法放入内存）

How to read multiple .mat files (which are too large to fit in memory) using tensorflow dataset

python

pickle

tensorflow

tensorflow-datasets