如何从包含路径和标签的 txt 文件为 tensorflow 创建数据集？

Question

我正在尝试将 DomainNet 数据集加载到张量流数据集中。每个域包含两个 .txt 文件，分别用于训练和测试数据，其结构如下：

painting/aircraft_carrier/painting_001_000106.jpg 0
painting/aircraft_carrier/painting_001_000060.jpg 0
painting/aircraft_carrier/painting_001_000130.jpg 0
painting/aircraft_carrier/painting_001_000058.jpg 0
painting/aircraft_carrier/painting_001_000093.jpg 0
painting/aircraft_carrier/painting_001_000107.jpg 0
painting/aircraft_carrier/painting_001_000088.jpg 0
painting/aircraft_carrier/painting_001_000014.jpg 0
painting/aircraft_carrier/painting_001_000013.jpg 0
...

每个图像一行，包含相对路径和标签。我的问题是，如果 tensorflow/keras 中已经有一些内置的方式来加载这种结构，或者我是否必须手动解析和加载数据？到目前为止，我的 google-fu 让我失望了...

Answer 1

您可以使用tf.data.TextLineDataset一次加载和处理多个txt文件：

import tensorflow as tf
import matplotlib.pyplot as plt

with open('data.txt', 'w') as f:
  f.write('/content/result_image1.png 0\n')
  f.write('/content/result_image2.png 1\n')

with open('more_data.txt', 'w') as f:
  f.write('/content/result_image1.png 1\n')
  f.write('/content/result_image2.png 0\n')

dataset = tf.data.TextLineDataset(['/content/data.txt', '/content/more_data.txt'])
for element in dataset.as_numpy_iterator():
  print(element)

b'/content/result_image1.png 0'
b'/content/result_image2.png 1'
b'/content/result_image1.png 1'
b'/content/result_image2.png 0'

处理数据：

def process(x):
  splits = tf.strings.split(x, sep=' ')
  image_path, label = splits[0], splits[1]
  img = tf.io.read_file(image_path)
  img = tf.io.decode_png(img, channels=3)
  return  img, tf.strings.to_number(label, out_type=tf.int32)

dataset = dataset.map(process)
for x, y in dataset.take(1):
  print('Label -->', y)
  plt.imshow(x.numpy())

Label --> tf.Tensor(0, shape=(), dtype=int32)

如何从包含路径和标签的 txt 文件为 tensorflow 创建数据集？

How to create a dataset for tensorflow from a txt file containing paths and labels?

python

dataset

tensorflow

tensorflow-datasets