如何根据 Tensorflow.js 中的边界框裁剪人脸？

Question

我需要裁剪在 BlazeFace 模型中检测到的人脸，然后将图像发送到我制作的自定义模型。我已经使用边界框实现了人脸检测，但仍然无法裁剪脸部。

我有地标和 bottomRight 和 topLeft 的坐标，但我不知道该怎么做。在 python 和 tensorflow 中，它们存在这样做的功能，但是在 tensorflow.js 我找不到任何东西。

在面部渲染边界框

    const faces = await bfModel
      .estimateFaces(tensor, returnTensors)
      .catch(e => console.log(e));
    console.log(faces);

    // Faces is an array of objects
    if (!isEmpty(faces)) {
      setModelFaces({ faces });
    }

  const renderBoundingBoxes = () => {
    const { faces } = modelFaces;
    const scale = {
      height: styles.camera.height / tensorDims.height,
      width: styles.camera.width / tensorDims.width
    };
    const flipHorizontal = Platform.OS === "ios" ? false : true;
    if (!isEmpty(faces)) {
      return faces.map((face, i) => {
        const { topLeft, bottomRight } = face;
        const bbLeft = topLeft.dataSync()[0] * scale.width;
        const boxStyle = Object.assign({}, styles.bbox, {
          left: flipHorizontal
            ? previewWidth - bbLeft - previewLeft
            : bbLeft + previewLeft,
          top: topLeft.dataSync()[1] * scale.height + 20,
          width:
            (bottomRight.dataSync()[0] - topLeft.dataSync()[0]) * scale.width,
          height:
            (bottomRight.dataSync()[1] - topLeft.dataSync()[1]) * scale.height
        });

        return <View style={boxStyle}></View>;
        1;
      });
    }
  };

console.log（面）的输出：

Array [
  Object {
    "bottomRight": Tensor {
      "dataId": Object {},
      "dtype": "float32",
      "id": 220600,
      "isDisposedInternal": false,
      "kept": false,
      "rankType": "1",
      "scopeId": 426282,
      "shape": Array [
        2,
      ],
      "size": 2,
      "strides": Array [],
    },
    "landmarks": Tensor {
      "dataId": Object {},
      "dtype": "float32",
      "id": 220602,
      "isDisposedInternal": false,
      "kept": false,
      "rankType": "2",
      "scopeId": 426286,
      "shape": Array [
        6,
        2,
      ],
      "size": 12,
      "strides": Array [
        2,
      ],
    },
    "probability": Tensor {
      "dataId": Object {},
      "dtype": "float32",
      "id": 220592,
      "isDisposedInternal": false,
      "kept": false,
      "rankType": "1",
      "scopeId": 426249,
      "shape": Array [
        1,
      ],
      "size": 1,
      "strides": Array [],
    },
    "topLeft": Tensor {
      "dataId": Object {},
      "dtype": "float32",
      "id": 220599,
      "isDisposedInternal": false,
      "kept": false,
      "rankType": "1",
      "scopeId": 426280,
      "shape": Array [
        2,
      ],
      "size": 2,
      "strides": Array [],
    },
  },
]

Answer 1

可以使用 tf.image.cropAndResize 裁剪图像。张量应该是 4d 张量。如果图像是 3d 张量，首先需要对其进行扩展。作物预期的高度和宽度应作为参数传递给 copAndResize

boxes = tf.concat([topLeftTensor, bottomRightTensor]).reshape([-1, 4])
crop = tf.image.cropAndResize(images, boxes, [0], [height, width])

Answer 2

我不确定，但它可以工作：

let img_4d = tf.expandDims(inputTF3D, axis=0)
const start = predictions[id].topLeft;
const end = predictions[id].bottomRight;
let boxes = tf.concat([start, end]);
boxes = tf.reshape(boxes,[-1, 4])
let crop = tf.image.cropAndResize(img_4d, boxes, [0], [IMAGE_HEIGHT, IMAGE_WIDTH])

https://github.com/tensorflow/tfjs/issues/3914

如何根据 Tensorflow.js 中的边界框裁剪人脸？

How do I crop faces based on bounding boxes in Tensorflow.js?

reactjs

deep-learning

react-native

tensorflow

tensorflow.js