实现 AVVideoCompositing 导致视频旋转问题

Question

我使用 Apple 的示例 https://developer.apple.com/library/ios/samplecode/AVCustomEdit/Introduction/Intro.html，但在视频转换方面存在一些问题。如果源资产具有 preferredTransform 而不是标识，则输出视频将具有错误旋转的帧。如果 AVMutableVideoComposition 在属性 customVideoCompositorClass 中没有值并且当 AVMutableVideoCompositionLayerInstruction 的转换设置为 asset.preferredTransform 时，可以解决此问题。但是由于使用采用 AVVideoCompositing 协议的自定义视频合成器，我无法使用标准视频合成指令。在将 CVPixelBuffer 放入 Metal 着色器之前，如何预转换输入资产轨道？或者还有其他方法可以解决吗？

原代码片段：

func buildCompositionObjectsForPlayback(_ forPlayback: Bool, overwriteExistingObjects: Bool) {

    // Proceed only if the composition objects have not already been created.
    if self.composition != nil && !overwriteExistingObjects { return }
    if self.videoComposition != nil && !overwriteExistingObjects { return }

    guard !clips.isEmpty else { return }

    // Use the naturalSize of the first video track.
    let videoTracks = clips[0].tracks(withMediaType: AVMediaType.video)
    let videoSize = videoTracks[0].naturalSize

    let composition = AVMutableComposition()

    composition.naturalSize = videoSize

    /*
     With transitions:
     Place clips into alternating video & audio tracks in composition, overlapped by transitionDuration.
     Set up the video composition to cycle between "pass through A", "transition from A to B", "pass through B".
    */
    let videoComposition = AVMutableVideoComposition()

    if self.transitionType == TransitionType.diagonalWipe.rawValue {
        videoComposition.customVideoCompositorClass = APLDiagonalWipeCompositor.self
    } else {
        videoComposition.customVideoCompositorClass = APLCrossDissolveCompositor.self
    }

    // Every videoComposition needs these properties to be set:
    videoComposition.frameDuration = CMTimeMake(1, 30) // 30 fps.
    videoComposition.renderSize = videoSize

    buildTransitionComposition(composition, andVideoComposition: videoComposition)

    self.composition = composition
    self.videoComposition = videoComposition
}

更新： 我做了像这样转换的解决方法：

private func makeTransformedPixelBuffer(fromBuffer buffer: CVPixelBuffer, withTransform transform: CGAffineTransform) -> CVPixelBuffer? {
    guard let newBuffer = renderContext?.newPixelBuffer() else {
        return nil
    }

    // Correct transformation example I took from 
    var preferredTransform = transform
    preferredTransform.b *= -1
    preferredTransform.c *= -1

    var transformedImage = CIImage(cvPixelBuffer: buffer).transformed(by: preferredTransform)
    preferredTransform = CGAffineTransform(translationX: -transformedImage.extent.origin.x, y: -transformedImage.extent.origin.y)
    transformedImage = transformedImage.transformed(by: preferredTransform)

    let filterContext = CIContext(mtlDevice: MTLCreateSystemDefaultDevice()!)
    filterContext.render(transformedImage, to: newBuffer)

    return newBuffer
}

但想知道是否有更多内存有效的方法而不创建新的像素缓冲区

Answer 1

How can I pre-transform input asset tracks before it's CVPixelBuffer's putted into Metal shaders?

获得最佳性能的最佳方法是直接在着色器中转换视频帧。您只需要在顶点着色器中添加旋转矩阵。

实现 AVVideoCompositing 导致视频旋转问题

Implementing AVVideoCompositing causes video rotation problems

core-image

avfoundation

core-video

swift

metal