视觉框架为捕获的视频输出帧中的矩形提供了错误的坐标

Question

我需要从捕获的视频中识别帧中的矩形。我使用以下方法在观察图像的顶部显示一个矩形。

func displayRect(for observation: VNRectangleObservation) {
    DispatchQueue.main.async { [weak self] in
        guard let size = self?.imageView.frame.size else { return }
        guard let origin = self?.imageView.frame.origin else { return }

        let transform = CGAffineTransform(scaleX: size.width, y: size.height)

        let rect = observation.boundingBox.applying(transform)
            .applying(CGAffineTransform(scaleX: 1.0, y: -1.0))
            .applying(CGAffineTransform(translationX: 0.0, y: size.height))
            .applying(CGAffineTransform(translationX: -origin.x, y: -origin.y))

        let path = UIBezierPath(rect: rect)

        let layer = CAShapeLayer()
        layer.path = path.cgPath
        layer.fillRule = kCAFillRuleEvenOdd
        layer.fillColor = UIColor.red.withAlphaComponent(0.2).cgColor

        self?.overlay.sublayers = nil
        self?.overlay.addSublayer(layer)
    }
}

这对于从相机拍摄的图像来说效果很好，但对于来自拍摄视频的帧，矩形是关闭的。事实上，如果偏离 90 度，它看起来就像它（以及图像的整个坐标系）。请看下面的截图。

我是否遗漏了有关视频帧的某些内容，可能会导致观察结果的 boundingBox 属性处于完全不同的坐标系中？

下面是我对 captureOutput 委托方法的实现。

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    guard let buffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }

    // Also tried converting to CGImage, creating handler from that, but made no difference
    let handler = VNImageRequestHandler(cvPixelBuffer: buffer, options: [:])

    let request = VNDetectRectanglesRequest()
    request.minimumAspectRatio = VNAspectRatio(0.2)
    request.maximumAspectRatio = VNAspectRatio(1.0)
    request.minimumSize = Float(0.3)

    try? handler.perform([request])

    // Note: Only ever captures one rectangle, so calling `first` not the issue.
    guard let observations = request.results as? [VNRectangleObservation],
        let observation = observations.first else {
            return removeShapeLayer()
    }

    displayRect(for: observation, buffer: buffer)
}

Answer 1

这个问题是因为您没有将缓冲区的方向传递给 VNImageRequestHandler，所以它将视频作为横向进行交易。然后当它 return 那个矩形时，你把它放在纵向显示的视频上方。

您需要将方向传递给 VNImageRequestHandler，或者修改（旋转）矩形 returned 以将其考虑在内。

视觉框架为捕获的视频输出帧中的矩形提供了错误的坐标

Vision framework gives incorrect coordinates for rectangles in frames from captured video output

ios

avcapturesession

swift