使用 PyTorch 和 TorchVision 对自定义数据集进行训练-有效-测试拆分

Question

我有一些用于二元分类任务的图像数据，这些图像被组织到 2 个文件夹中，分别是 data/model_data/class-A 和 data/model_data/class-B。

共有N张图片。我想要 train/val/test 的比例为 70/20/10。我正在使用 PyTorch 和 Torchvision 来完成这项任务。这是我目前的代码。

from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils, datasets, models

data_transform = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])

model_dataset = datasets.ImageFolder(root, transform=data_transform) 
train_count = int(0.7 * total_count) 
valid_count = int(0.2 * total_count)
test_count = total_count - train_count - valid_count
train_dataset, valid_dataset, test_dataset = torch.utils.data.random_split(model_dataset, (train_count, valid_count, test_count))
train_dataset_loader = torch.utils.data.DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKER)  
valid_dataset_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKER) 
test_dataset_loader  = torch.utils.data.DataLoader(test_dataset , batch_size=BATCH_SIZE, shuffle=False,num_workers=NUM_WORKER)
dataloaders = {'train': train_dataset_loader, 'val': valid_dataset_loader, 'test': test_dataset_loader}

我觉得这不是正确的做法，原因有二。

我正在对所有拆分应用相同的转换。（这显然不是我想要做的！解决这个问题的答案很可能是。）
通常人们首先将原始数据分离成test/train然后他们将火车分成train/val，而我直接将原始数据转化为train/val/test。（这是正确的吗？）

那么，我的问题是，我的做法是否正确？（可能不是）
如果不正确，我该如何编写数据加载器来实现所需的拆分，以便我可以对每个 train/test/val?

应用单独的转换

Answer 1

Usually people first separate the original data into test/train and then they separate train into train/val, whereas I am directly separating the original data into train/val/test. (Is this correct?)

是的，它是完全正确的，可读性强，而且完全没问题

I am applying the same transform to all the splits. (This is not what I want to do, obviously! The solution for this is most probably the answer here.)

是的，这个答案是可能的，但它毫无意义地冗长。您可以使用第三方工具 torchdata，只需安装：

pip install torchdata

可以找到文档 here（同时免责声明：我是作者）。

它允许您轻松地将转换映射到任何 torch.utils.data.Dataset（在本例中为 train）。您的代码看起来像这样（只需更改两行，检查注释，还格式化您的代码以使其更容易遵循）：

import torch
import torchvision

import torchdata as td

data_transform = torchvision.transforms.Compose(
    [
        torchvision.transforms.RandomResizedCrop(224),
        torchvision.transforms.RandomHorizontalFlip(),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize(
            mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]
        ),
    ]
)

# Single change, makes an instance of torchdata.Dataset
# Works just like PyTorch's torch.utils.data.Dataset, but has
# additional capabilities like .map, cache etc., see project's description
model_dataset = td.datasets.WrapDataset(torchvision.datasets.ImageFolder(root))
# Also you shouldn't use transforms here but below
train_count = int(0.7 * total_count)
valid_count = int(0.2 * total_count)
test_count = total_count - train_count - valid_count
train_dataset, valid_dataset, test_dataset = torch.utils.data.random_split(
    model_dataset, (train_count, valid_count, test_count)
)

# Apply transformations here only for train dataset

train_dataset = train_dataset.map(data_transform)

# Rest of the code goes the same

train_dataset_loader = torch.utils.data.DataLoader(
    train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKER
)
valid_dataset_loader = torch.utils.data.DataLoader(
    valid_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKER
)
test_dataset_loader = torch.utils.data.DataLoader(
    test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=NUM_WORKER
)
dataloaders = {
    "train": train_dataset_loader,
    "val": valid_dataset_loader,
    "test": test_dataset_loader,
}

是的，我同意在拆分之前指定 transform 不太清楚，IMO 这样更易读。

使用 PyTorch 和 TorchVision 对自定义数据集进行训练-有效-测试拆分

Train-Valid-Test split for custom dataset using PyTorch and TorchVision

python

pytorch

torchvision