无法使用 AVAssetTrackReaderOutput 在 while 循环中更新 UIViewImage

Question

我正在尝试从视频中提取每一帧进行一些图像处理，然后将图像显示回 UIImageView。

如果我手动单击 UIButton，图像将显示并遍历整个视频并显示每一帧。

但是，如果我在 while 循环中嵌入更新显示，则视图不会更新（即，在设备上它不会更新）。

我认为这是因为相对于在屏幕上绘制更新的图像而言，帧的处理速度太快，所以我放入了睡眠线以将其减慢到视频的帧速率，但是那无效。

代码如下：

import UIKit
import Foundation
import Vision
import AVFoundation
import Darwin

class ViewController: UIViewController {

    var uiImage: UIImage?

    var displayLink: CADisplayLink?

    var videoAsset: AVAsset!
    var videoTrack: AVAssetTrack!
    var videoReader: AVAssetReader!
    var videoOutput: AVAssetReaderTrackOutput!


    @IBOutlet weak var topView: UIImageView!

    @IBOutlet weak var bottomView: UIImageView!

    @IBOutlet weak var rightLabel: UILabel!

    @IBOutlet weak var leftLabel: UILabel!

    @IBAction func tapButton(_ sender: Any) {

        while let sampleBuffer = videoOutput.copyNextSampleBuffer() {
            print ("sample at time \(CMSampleBufferGetPresentationTimeStamp(sampleBuffer))")
            if let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {

                let ciImage = CIImage(cvImageBuffer: imageBuffer)
                uiImage = UIImage(ciImage: ciImage)
                self.topView.image = uiImage
                self.topView.setNeedsDisplay()
                usleep(useconds_t(24000))

            }

        }
    }


    override func viewDidLoad() {

        super.viewDidLoad()

    }



    override func viewDidAppear(_ animated: Bool) {




        guard let urlPath = Bundle.main.path(forResource: "video1", ofType: "mp4")  else {
            print ("No File")
            return
        }

        videoAsset = AVAsset(url: URL(fileURLWithPath: urlPath))
        let array = videoAsset.tracks(withMediaType: AVMediaType.video)
        videoTrack = array[0]


        do {
            videoReader = try AVAssetReader(asset: videoAsset)
        } catch {
            print ("No reader created")
        }

        videoOutput = AVAssetReaderTrackOutput(track: videoTrack, outputSettings: [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange])
        videoReader.add(videoOutput)
        videoReader.startReading()


    }

}

Answer 1

在你的情况下，你应该使用计时器而不是睡觉。尝试如下操作：

let frameDuration: TimeInterval = 1.0/60.0 // Using 60 FPS
Timer.scheduledTimer(withTimeInterval: frameDuration, repeats: true) { timer in
    guard let sampleBuffer = videoOutput.copyNextSampleBuffer() else  {
        timer.invalidate()
        return
    }

    print ("sample at time \(CMSampleBufferGetPresentationTimeStamp(sampleBuffer))")
    if let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {
        let ciImage = CIImage(cvImageBuffer: imageBuffer)
        uiImage = UIImage(ciImage: ciImage)
        self.topView.image = uiImage
        self.topView.setNeedsDisplay()
    }
}

你的问题是你在执行此操作时仍在主线程上。所以当你睡觉时，你不会让你的主线程继续进行。所以基本上你的主线程正在构建图像和休眠，但你没有给它时间来实际更新用户界面。

您可以使用单独的线程做类似的事情。例如，一个简单的调度就可以做到：

DispatchQueue(label: "Updating images").async {
    // Now on separate thread
    while let sampleBuffer = videoOutput.copyNextSampleBuffer() {
        print ("sample at time \(CMSampleBufferGetPresentationTimeStamp(sampleBuffer))")
        if let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {
            let ciImage = CIImage(cvImageBuffer: imageBuffer)
            uiImage = UIImage(ciImage: ciImage)

            // The UI part now needs to go back to main thread
            DispatchQueue.main.async {
                self.topView.image = uiImage
                self.topView.setNeedsDisplay()
            }

            usleep(useconds_t(24000))

        }
    }
}

但重要的是您仍然（至少）在主线程上更新 UIKit 部分。可能甚至 CI 部分也需要在主线程上。最好只是尝试一下。

无法使用 AVAssetTrackReaderOutput 在 while 循环中更新 UIViewImage

Cannot update UIViewImage in while loop with AVAssetTrackReaderOutput

avfoundation

uikit

ios

swift