如何在捕获会话中将 CIFilter 与 CIPerspectiveCorrection 结合使用？

Question

我想用 phone 扫描文档并修复任何透视问题，就像笔记应用程序一样。一切都很好，直到我想使用 CIFilter(name: "CIPerspectiveCorrection")，然后我把图像弄乱了，我很难理解我哪里出错了。

我尝试过切换参数和其他滤镜，或旋转图像，但对我来说不起作用。

这是我设置的一个小项目来测试这一切： https://github.com/iViktor/scanner

基本上我是运行在 AVCaptureSession 上设置 VNDetectRectanglesRequest 并保存我在 private var targetRectangle = VNRectangleObservation()

中得到的矩形

我用来重新计算图像内的点并运行图像上的过滤器的那个。

extension DocumentScannerViewController: AVCapturePhotoCaptureDelegate {
func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
    guard let imageData = photo.fileDataRepresentation()
        else { return }
    guard let ciImage = CIImage(data: imageData, options: [.applyOrientationProperty : true]) else { return }
    let image = UIImage(ciImage: ciImage)

    let imageTopLeft: CGPoint = CGPoint(x: image.size.width * targetRectangle.bottomLeft.x, y: targetRectangle.bottomLeft.y * image.size.height)
    let imageTopRight: CGPoint = CGPoint(x: image.size.width * targetRectangle.bottomRight.x, y: targetRectangle.bottomRight.y * image.size.height)
    let imageBottomLeft: CGPoint = CGPoint(x: image.size.width * targetRectangle.topLeft.x, y: targetRectangle.topLeft.y * image.size.height)
    let imageBottomRight: CGPoint = CGPoint(x: image.size.width * targetRectangle.topRight.x, y: targetRectangle.topRight.y * image.size.height)

    let flattenedImage = image.flattenImage(topLeft: imageTopLeft, topRight: imageTopRight, bottomLeft: imageBottomLeft, bottomRight: imageBottomRight)
    let finalImage = UIImage(ciImage: flattenedImage, scale: image.scale, orientation: image.imageOrientation)

//performSegue(withIdentifier: "showPhoto", sender: image)
//performSegue(withIdentifier: "showPhoto", sender: UIImage(ciImage: flattenedImage))
    performSegue(withIdentifier: "showPhoto", sender: finalImage)

}
}

这是无效的代码，我正在努力解决：

extension UIImage {

func flattenImage(topLeft: CGPoint, topRight: CGPoint, bottomLeft: CGPoint, bottomRight: CGPoint) -> CIImage {
    let docImage = self.ciImage!
    let rect = CGRect(origin: CGPoint.zero, size: self.size)
    let perspectiveCorrection = CIFilter(name: "CIPerspectiveCorrection")!
    perspectiveCorrection.setValue(CIVector(cgPoint: self.cartesianForPoint(point: topLeft, extent: rect)), forKey: "inputTopLeft")
    perspectiveCorrection.setValue(CIVector(cgPoint: self.cartesianForPoint(point: topRight, extent: rect)), forKey: "inputTopRight")
    perspectiveCorrection.setValue(CIVector(cgPoint: self.cartesianForPoint(point: bottomLeft, extent: rect)), forKey: "inputBottomLeft")
    perspectiveCorrection.setValue(CIVector(cgPoint: self.cartesianForPoint(point: bottomRight, extent: rect)), forKey: "inputBottomRight")
    perspectiveCorrection.setValue(docImage, forKey: kCIInputImageKey)

    return perspectiveCorrection.outputImage!
}

func cartesianForPoint(point:CGPoint,extent:CGRect) -> CGPoint {
    return CGPoint(x: point.x,y: extent.height - point.y)
}
}

所以最后我想扫描一份文件，比如发票，并自动修复任何用户错误，比如透视问题。现在，我添加到图像中的过滤器会产生类似手扇的奇怪效果。

Answer 1

根据评论，我更新了我使用 targetRectangle 的代码，改为使用绘制路径表示的点，并更改了我将它们用于图像的位置，因为 CI 使用不同的坐标系和图片是镜像的。

我更新了

    private func startScanner() {
         ... ... ...
               let request = VNDetectRectanglesRequest { req, error in
                    DispatchQueue.main.async {
                        if let observation = req.results?.first as? VNRectangleObservation {
                            let points = self.targetRectLayer.drawTargetRect(observation: observation, previewLayer: self.previewLayer, animated: false)
                            let size = self.scannerView.frame.size
                            self.trackedTopLeftPoint = CGPoint(x: points.topLeft.x / size.width, y: points.topLeft.y / size.height )
                            self.trackedTopRightPoint = CGPoint(x: points.topRight.x / size.width, y: points.topRight.y / size.height )
                            self.trackedBottomLeftPoint = CGPoint(x: points.bottomLeft.x / size.width, y: points.bottomLeft.y / size.height )
                            self.trackedBottomRightPoint = CGPoint(x: points.bottomRight.x / size.width, y: points.bottomRight.y / size.height )
                        } else {
                            _ = self.targetRectLayer.drawTargetRect(observation: nil, previewLayer: self.previewLayer, animated: false)
                        }
                    }
                }
        }

和

extension DocumentScannerViewController: AVCapturePhotoCaptureDelegate {
func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
    guard let imageData = photo.fileDataRepresentation()
        else { return }
    guard let ciImage = CIImage(data: imageData, options: [.applyOrientationProperty : true]) else { return }
    let image = UIImage(ciImage: ciImage)

    // CoreImage is working with cartesian coordinates, basically y:0 is in the bottom left corner
    let imageTopLeft: CGPoint = CGPoint(x: image.size.width * trackedBottomLeftPoint.x, y: trackedBottomLeftPoint.y * image.size.height)
    let imageTopRight: CGPoint = CGPoint(x: image.size.width * trackedTopLeftPoint.x, y: trackedTopLeftPoint.y * image.size.height)
    let imageBottomLeft: CGPoint = CGPoint(x: image.size.width * trackedBottomRightPoint.x, y: trackedBottomRightPoint.y * image.size.height)
    let imageBottomRight: CGPoint = CGPoint(x: image.size.width * trackedTopRightPoint.x, y: trackedTopRightPoint.y * image.size.height)

    let flattenedImage = image.flattenImage(topLeft: imageTopLeft, topRight: imageTopRight, bottomLeft: imageBottomLeft, bottomRight: imageBottomRight)
    let newCGImage = CIContext(options: nil).createCGImage(flattenedImage, from: flattenedImage.extent)
    let doneCroppedImage = UIImage(cgImage: newCGImage!, scale: image.scale, orientation: image.imageOrientation)
    performSegue(withIdentifier: "showPhoto", sender: doneCroppedImage)
}
}

已修复。

如何在捕获会话中将 CIFilter 与 CIPerspectiveCorrection 结合使用？

How to use CIFilter with CIPerspectiveCorrection on a capture session?

swift

core-image