Serialize/deserialize FiftyOne App 中的图像遮罩

Question

我想使用 FiftyOne 应用程序来评估实例分割预测。我使用可下载的快速入门数据集开始，并为地面实况和预测数据创建了自己的 samples.json 文件，可以通过 Dataset.from_dir 加载。教程中的 json 文件仅包含边界框，但不包含遮罩。在文档中，我只找到了设置为 None 的掩码，但我需要它们。

{
  "samples": [
    {
      "filepath": "some_image.png",
      "tags": [
        "validation"
      ],
      "metadata": null,
      "uniqueness": 0.998,
      "ground_truth": {
        "_cls": "Detections",
        "detections": [
          {
            "_id": {
              "$oid": "efd81c917c9a49dc9f8aca3f"
            },
            "_cls": "Detection",
            "label": "1",
            "bounding_box": [
              0.62609375,
              0.27044921875,
              0.030078124999999956,
              0.009550781250000001
            ],
            "mask": ???,
          }
        ]
      },
      "predictions": {
        "_cls": "Detections",
        "detections": [
          {
            "_id": {
              "$oid": "6db44c71c1414b8989c92255"
            },
            "_cls": "Detection",
            "label": "1",
            "bounding_box": [
              0.3303889036178589,
              0.4432219862937927,
              0.07914596796035767,
              0.02226179838180542
            ],
            "mask": ???,
            "confidence": 0.999962329864502
          },
        ]
      }
    }
  ]
}

我遇到的问题是，我应该如何为分割掩码创建 mask 属性？它是一个 numpy 数组，默认情况下不可序列化。 Dataset.from_dir 使用 core.utils.deserialize_numpy_array 函数加载数据集，所以我尝试使用 serialize_numpy_array 保存数据集但没有成功。

那么将掩码写入可反序列化的 json 文件的最佳方法是什么？谢谢！

Answer 1

Dataset.from_dir() 语法通常用于 well-defined 数据集类型，例如以 COCODetectionDataset 格式存储在磁盘上的数据集。

在您的情况下，当加载不直接对应于相关数据集类型之一的自定义数据集时，建议的方法是编写 simple Python loop and construct your dataset one sample at a time.

在这种情况下，您只需要 load your mask as a numpy array and store it in the mask attribute of your detection object。

import glob
import fiftyone as fo

images_patt = "/path/to/images/*"

# Ex: your custom label format
annotations = {
    "/path/to/images/000001.jpg": [
        {"bbox": ..., "label": ..., "mask": ...},
        ...
    ],
    ...
}

# Create samples for your data
samples = []
for filepath in glob.glob(images_patt):
    sample = fo.Sample(filepath=filepath)

    # Convert detections to FiftyOne format
    detections = []
    for objects in annotations[filepath]:
        label = obj["label"]

        # Bounding box coordinates should be relative values
        # in [0, 1] in the following format:
        # [top-left-x, top-left-y, width, height]
        bounding_box = obj["bbox"]

        # Boolean or 0/1 Numpy array 
        mask = obj["mask"]
        
        detection = fo.Detection(
            label=label,
            bounding_box=bounding_box,
            mask=mask,
        )

        detections.append(detection)

    # Store detections in a field name of your choice
    sample["ground_truth"] = fo.Detections(detections=detections)

    samples.append(sample)

# Create dataset
dataset = fo.Dataset("my-detection-dataset")
dataset.add_samples(samples)

预测可以与基本事实同时加载，或者您以后可以随时迭代您的数据集 add predictions at a later time:

import fiftyone as fo

# Ex: your custom predictions format
predictions = {
    "/path/to/images/000001.jpg": [
        {"bbox": ..., "label": ..., "mask": ..., "score": ...},
        ...
    ],
    ...
}

# Add predictions to your samples
for sample in dataset:
    filepath = sample.filepath

    # Convert predictions to FiftyOne format
    detections = []
    for obj in predictions[filepath]:
        label = obj["label"]
        confidence = obj["score"]

        # Bounding box coordinates should be relative values
        # in [0, 1] in the following format:
        # [top-left-x, top-left-y, width, height]
        bounding_box = obj["bbox"]
        
        # Boolean or 0/1 Numpy array 
        mask = obj["mask"]

        detection = fo.Detection(
            label=label,
            bounding_box=bounding_box,
            confidence=confidence,
        )

        detection["mask"] = mask

        detections.append(detection)

    # Store detections in a field name of your choice
    sample["predictions"] = fo.Detections(detections=detections)

    sample.save()

请注意，如何解析自定义标签格式的确切结构将取决于您存储数据的方式。这只是将标签存储在由媒体文件路径键入的字典中的一个示例。您可能首先需要解析和转换您的标签，例如将其转换为预期的边界框格式。

Serialize/deserialize FiftyOne App 中的图像遮罩

Serialize/deserialize image mask in FiftyOne App

json

object-detection

fiftyone