IOS ML Kit 面部跟踪无法正常工作

Question

人脸检测的tracking Id在人脸不动的情况下一直在变化，我在ios中使用了ML Kit，并遵循了google的文档。

文档：https://developers.google.com/ml-kit/vision/face-detection/ios#performance_tips

这是我的代码：

func captureOutput(  _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
         self.detectFaceMLKit(image: sampleBuffer)
}

private func detectFaceMLKit(image: CMSampleBuffer) {
        let image = VisionImage(buffer: image)
        image.orientation = imageOrientation(
            deviceOrientation: UIDevice.current.orientation,
            cameraPosition: cameraPosition)
               
        faceDetector?.process(image) { [self] faces, error in
          guard error == nil, let faces = faces, !faces.isEmpty else {
            //TODO : show error
            return
          }

            // Faces detected
            if faces.count == 1 {
                let face = faces[0]
                let frame = face.frame
                var rotX : CGFloat = 0
                var rotY : CGFloat = 0
                var faceTrackingId = 0
                if face.hasHeadEulerAngleX {
                    rotX = face.headEulerAngleX  // Head is rotated to the uptoward rotX degrees
                    print("rotx: \(rotX)")
                }
                if face.hasHeadEulerAngleY {
                    rotY = face.headEulerAngleY  // Head is rotated to the right rotY degrees
                    print("rotY: \(rotY)")
                }
                // If face tracking was enabled:
                if face.hasTrackingID {
                    faceTrackingId = face.trackingID
                    print("trackingId: \(faceTrackingId)")
                }
    }
}


    func imageOrientation(
        deviceOrientation: UIDeviceOrientation,
        cameraPosition: AVCaptureDevice.Position
    ) -> UIImage.Orientation {
        switch deviceOrientation {
        case .portrait:
            return cameraPosition == .front ? .leftMirrored : .right
        case .landscapeLeft:
            return cameraPosition == .front ? .downMirrored : .up
        case .portraitUpsideDown:
            return cameraPosition == .front ? .rightMirrored : .left
        case .landscapeRight:
            return cameraPosition == .front ? .upMirrored : .down
        case .faceDown, .faceUp, .unknown:
            return .up
        }
    }

注意：跟踪 ID 在开始时工作正常，但在 ID“10”附近它失去面部跟踪并开始为同一张脸提供多 ID，而无需在相机中移动面部。

过滤跟踪 ID 后的输出样本：

trackingId: 21 trackingId: 21 trackingId: 21 trackingId: 21 trackingId: 21 trackingId: 21 trackingId: 22 trackingId: 22 trackingId: 22 trackingId: 22 trackingId: 22 trackingId: 23 trackingId: 23 trackingId: 23 trackingId: 23 trackingId: 23 trackingId: 24 trackingId: 24 trackingId: 24 trackingId: 24 trackingId: 24

Answer 1

问题是 imageOrientation，我只在 Xcode 中将方向设置为纵向，但在 UIDeviceOrientation 上旋转 image-based 这是错误的，通过将 imageOrientation 设置为固定在 .up 来修复它位置。

编辑：另外，请确保您没有像这样覆盖输出图像方向：

 private let videoDataOutput = AVCaptureVideoDataOutput()
 guard let connection = self.videoDataOutput.connection(with: AVMediaType.video),
            connection.isVideoOrientationSupported else { return }
 connection.videoOrientation = .portrait

IOS ML Kit 面部跟踪无法正常工作

IOS ML Kit face tracking does not work correctly

xcode

ios

swift

firebase-mlkit

google-mlkit