将 ProresRAW 格式解码为原生拜耳表示法

Question

我正在尝试解码 Prores 视频文件，但它不起作用。我总是

Optional(Error Domain=AVFoundationErrorDomain Code=-11821 "Cannot Decode" UserInfo={NSLocalizedFailureReason=媒体数据无法解码，可能已损坏，NSLocalizedDescription=无法解码，NSUnderlyingError=0x600002a982a0 {Error Domain= NSOSStatusErrorDomain Code=-12137 "(null)"}})

这里是一个完整的解码器：

class Decoder {

private let assetReader: AVAssetReader?
private let output: AVAssetReaderTrackOutput

init() throws {
    
    VTRegisterProfessionalVideoWorkflowVideoDecoders()
    VTRegisterProfessionalVideoWorkflowVideoEncoders()
    
        let assetReader = try AVAssetReader(asset: movieAsset)
        let tracks = movieAsset.tracks(withMediaType: .video)
        
        guard let firstTrack = tracks.first else {
            print("No video tracks found")
            throw NSError()
        }
    
        let out = AVAssetReaderTrackOutput(track: firstTrack, outputSettings: outputSettings)
        out.alwaysCopiesSampleData = true
        
        assetReader.add(out)
        
        self.assetReader = assetReader
        self.output = out

   
}

func run(){
    
    guard let assetReader = assetReader, assetReader.startReading() else {
        print("Failed to stard asset reader")
        return
    }
    

    while(assetReader.status == .reading) {
        guard let sampleBuffer = output.copyNextSampleBuffer() else {
            print(assetReader.status.rawValue)
            print(assetReader.error)
            continue
        }
          
        print("Decoding success!")
    }
}

}

Answer 1

不清楚你为什么想要拜耳，我不确定你所说的“本地”是什么意思，但我想你可能希望你的数据是

尽可能高的清晰度，或
以最自然/最有效/最少处理的格式
就拜耳别再问我问题了

所以我认为有两种可能。

如果您喜欢高清数据，请尝试将您的 AVAssetReaderTrackOutput 像素格式设置为 kCVPixelFormatType_444YpCbCr16VideoRange_16A_TriPlanar、kCVPixelFormatType_4444AYpCbCr16 或 kCVPixelFormatType_64RGBALE 或 AVAssetReaderTrackOutput。我认为 AVAssetReader 不会无故截断数据的可能性很大。

在使用 ProRes RAW 时，我对自然或高效表示一无所知，但如果您真的想要 Bayer 输出，可以将 outputSettings 设置为 nil 并使用 VTDecompressionSession 将原始样本缓冲区转换为 kCVPixelFormatType_16VersatileBayer（或 kCVPixelFormatType_64RGBAHalf、kCVPixelFormatType_128RGBAFloat，如果您仍然使用 AVAssetReader 出于某种原因不喜欢的高范围格式），但是 not kCVPixelFormatType_64RGBA_DownscaledProResRAW 因为这似乎不起作用。

无论如何，您可以稍微修改代码以解码为 kCVPixelFormatType_16VersatileBayer，如下所示：

import AVFoundation
import VideoToolbox

class Decoder {
    private let assetReader: AVAssetReader?
    private let output: AVAssetReaderTrackOutput
    private var decompressionSession: VTDecompressionSession!
        
    init() throws {
        let movieUrl = URL(fileURLWithPath: "/Users/xxxx/ProresRAW_Video.MOV")
        let movieAsset = AVAsset(url: movieUrl)
        
        do {
            let assetReader = try AVAssetReader(asset: movieAsset)
            let tracks = movieAsset.tracks(withMediaType: .video)
            
            guard let firstTrack = tracks.first else {
                print("No video tracks found")
                throw NSError()
            }
            
            let out = AVAssetReaderTrackOutput(track: firstTrack, outputSettings: nil)
            out.alwaysCopiesSampleData = true
            
            assetReader.add(out)
            
            self.assetReader = assetReader
            self.output = out
            
        } catch {
            print(error)
            throw error
        }
        
    }
    
    func run(){
        guard let assetReader = assetReader, assetReader.startReading() else {
            print("Failed to stard asset reader")
            return
        }
        
        while(assetReader.status == .reading) {
            guard let sampleBuffer = output.copyNextSampleBuffer() else {
                print(assetReader.status.rawValue)
                print(assetReader.error)
                continue
            }
            
            print("Decoding success! \(sampleBuffer)")
            
            if let formatDescription = CMSampleBufferGetFormatDescription(sampleBuffer) {
                if decompressionSession == nil {
                    let imageBufferAttributes: [CFString: Any] = [
                        kCVPixelBufferPixelFormatTypeKey: kCVPixelFormatType_16VersatileBayer
                    ]
                    var outputCallback = VTDecompressionOutputCallbackRecord(decompressionOutputCallback: { _, _, status, infoFlags, imageBuffer, presentationTimeStamp, presentationDuration in
                        assert(noErr == status)
                        print("decode callback status: \(status), bayer imageBuffer \(String(describing: imageBuffer)), flags: \(infoFlags), pts: \(presentationDuration), ptsd: \(presentationDuration)")
                    }, decompressionOutputRefCon: nil)
                    let status = VTDecompressionSessionCreate(allocator: nil, formatDescription: formatDescription, decoderSpecification: nil, imageBufferAttributes: imageBufferAttributes as CFDictionary, outputCallback: &outputCallback, decompressionSessionOut: &decompressionSession)
                    assert(noErr == status)
                }

                let status = VTDecompressionSessionDecodeFrame(decompressionSession, sampleBuffer: sampleBuffer, flags: [], frameRefcon: nil, infoFlagsOut: nil)
                assert(noErr == status)
            }
        }
    }
}

我不明白的是为什么 AVAssetReader 可能在幕后使用 VTDecompressionSession 不简单地让您首先请求 kCVPixelFormatType_16VersatileBayer。也许这是血腥的思想，或者也许没有意义？ p.s。你想做什么？

将 ProresRAW 格式解码为原生拜耳表示法

Decoding ProresRAW format to native bayer representation

avfoundation

video-processing

avkit

video-toolbox