TokBox：'consumeFrame' 使用修改后的像素缓冲区时崩溃

Question

我正在尝试将来自 AVFoundation 的实时视频源的像素缓冲区修改为通过 OpenTok's API 进行流式传输。但是每当我尝试这样做并通过 OpenTok 的 consumeFrame 提供它时，它就会崩溃。

我这样做是为了应用不同的实时视频效果（滤镜、贴纸等）。我试过用不同的方法转换 CGImage->CVPixelBuffer，但没有任何效果。

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    if !capturing || videoCaptureConsumer == nil {
        return
    }

    guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
        else {
            print("Error acquiring sample buffer")
            return
    }

    guard let videoInput = videoInput
        else {
            print("Capturer does not have a valid input")
            return
    }

    let time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
    videoFrame.clearPlanes()

    videoFrame.timestamp = time
    let height = UInt32(CVPixelBufferGetHeight(imageBuffer))
    let width = UInt32(CVPixelBufferGetWidth(imageBuffer))

    if width != captureWidth || height != captureHeight {
        updateCaptureFormat(width: width, height: height)
    }

    // This is where I convert CVImageBuffer->CIImage, modify it, turn it into CGImage, then CGImage->CVPixelBuffer
    guard let finalImage = makeBigEyes(imageBuffer) else { return }

    CVPixelBufferLockBaseAddress(finalImage, CVPixelBufferLockFlags(rawValue: 0))

    videoFrame.format?.estimatedCaptureDelay = 10
    videoFrame.orientation = .left

    videoFrame.clearPlanes()
    videoFrame.planes?.addPointer(CVPixelBufferGetBaseAddress(finalImage))

    delegate?.finishPreparingFrame(videoFrame)

    videoCaptureConsumer!.consumeFrame(videoFrame)

    CVPixelBufferUnlockBaseAddress(finalImage, CVPixelBufferLockFlags(rawValue: 0))
}

这是我的 CGImage->CVPixelBuffer 方法：

    func buffer(from image: UIImage) -> CVPixelBuffer? {
    let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue, kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
    var pixelBuffer : CVPixelBuffer?
    let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(image.size.width), Int(image.size.height), kCVPixelFormatType_32ARGB, attrs, &pixelBuffer)
    guard (status == kCVReturnSuccess) else {
        return nil
    }

    CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
    let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)

    let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
    let context = CGContext(data: pixelData, width: Int(image.size.width), height: Int(image.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)

    context?.translateBy(x: 0, y: image.size.height)
    context?.scaleBy(x: 1.0, y: -1.0)

    UIGraphicsPushContext(context!)
    image.draw(in: CGRect(x: 0, y: 0, width: image.size.width, height: image.size.height))
    UIGraphicsPopContext()
    CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))

    return pixelBuffer
}

我在第一帧出现这个错误：

* Terminating app due to uncaught exception 'NSRangeException', reason: '* -[NSConcretePointerArray pointerAtIndex:]: attempt to access pointer at index 1 beyond bounds 1'

如果您读到了这里，感谢您的阅读。我已经在这个问题上停留了一段时间，所以任何类型的指针都将不胜感激。谢谢。

Answer 1

由于您要将相机 (NV12) 帧转换为 RGB，因此需要在 videoFrame.format

上将 pixelFormat 设置为 OTPixelFormatARGB

TokBox：'consumeFrame' 使用修改后的像素缓冲区时崩溃

TokBox: 'consumeFrame' crashes when using a modified pixel buffer

swift

core-image

opentok

tokbox

cvpixelbuffer