irregular/varying 张量流中的批量大小？

Question

我有一个 tensorflow 数据集，我想对它进行批处理，以便 not 具有相同的大小 - 就像示例被分组到批次中，其大小由 [= =19=]向量个值而不是一个固定值。

有没有办法在 tensorflow 中做到这一点？

对于没有固定批次大小的网络，喂养不规则的批次会成为问题吗？

提前致谢！

Answer 1

答案是肯定的。 model.fit() 方法 allows to pass to it a generator 将生成随机大小的批次。

x_train_batches = ... # some list of data batches of uneven length 
y_train_batches = ... # some list of targets of SAME lengths as x_train_batches

def train_gen(x_train_batches, y_train_batches):
    i = 0
    num_batches = len(x_train_batches)
    while True:
        yield (x_train_batches[i%num_batches], y_train_batches[i%num_batches])
        i += 1

model.fit(train_gen(x_train_batches, y_train_batches), epochs=epochs, steps_per_epoch=NUM_BATCHES)

另一种更优雅的方法是继承 tf.keras.utils.Sequence 并将其提供给模型：

class UnevenSequence(keras.utils.Sequence):
      def __init__(self, x_batches, y_batches):
          # x_batches, y_batches are lists of uneven batches
          self.x, self.y = x_batches, y_batches
      def __len__(self):
          return len(self.x)
      def __getitem__(self, idx):
          batch_x = self.x[idx]
          batch_y = self.y[idx]
          return (batch_x, batch_y)

my_uneven_sequence = UnevenSequence(x_train_batches, y_train_batches)

model.fit(my_uneven_sequence, epochs=10)

irregular/varying 张量流中的批量大小？

irregular/varying batch size in tensorflow?

python

tensorflow

keras

tensorflow-datasets

batching