从 CMSampleBuffer 中提取数据以创建深拷贝

Question

我正在尝试创建 CMSampleBuffer 的副本，由 AVCaptureVideoDataOutputSampleBufferDelegate 中的 captureOutput 返回。

由于 CMSampleBuffers 来自预先分配的 (15) 个缓冲区池，如果我附加对它们的引用，它们将无法重新收集。这会导致所有剩余的帧被丢弃。

To maintain optimal performance, some sample buffers directly reference pools of memory that may need to be reused by the device system and other capture inputs. This is frequently the case for uncompressed device native capture where memory blocks are copied as little as possible. If multiple sample buffers reference such pools of memory for too long, inputs will no longer be able to copy new samples into memory and those samples will be dropped.

If your application is causing samples to be dropped by retaining the provided CMSampleBufferRef objects for too long, but it needs access to the sample data for a long period of time, consider copying the data into a new buffer and then releasing the sample buffer (if it was previously retained) so that the memory it references can be reused.

显然我必须复制 CMSampleBuffer，但 CMSampleBufferCreateCopy() 只会创建一个浅表副本。因此我得出结论，我必须使用 CMSampleBufferCreate()。我填的是12！构造函数需要的参数，但运行我的 CMSampleBuffers 不包含 blockBuffer 的问题（不完全确定那是什么，但它似乎很重要）。

这个问题已经被问过好几次了，但是没有人回答。

Deep Copy of CMImageBuffer or CVImageBuffer and

一个可能的答案是“我终于想出了如何使用它来创建深度克隆。所有的复制方法都重复使用了堆中的数据，这些数据会锁定 AVCaptureSession。所以我不得不将数据拉出到一个 NSMutableData 对象，然后创建一个新的样本缓冲区。” 。但是，我不知道如何正确地做到这一点。

有兴趣的话，this就是print(sampleBuffer)的输出。没有提到 blockBuffer，又名 CMSampleBufferGetDataBuffer returns nil。有一个 imageBuffer，但是使用 CMSampleBufferCreateForImageBuffer 创建“副本”似乎也没有释放 CMSampleBuffer。

编辑：自从发布了这个问题后，我一直在尝试更多复制内存的方法。

我做了同样的事情，用户 Kametrixom tried. This 是我尝试同样的想法，首先复制 CVPixelBuffer 然后使用 CMSampleBufferCreateForImageBuffer 创建最终的样本缓冲区。然而，这会导致以下两个错误之一：

A EXC_BAD_ACCESS 在 memcpy 指令上。也就是尝试访问应用程序内存之外的段错误。
或者，内存将成功复制，但 CMSampleBufferCreateReadyWithImageBuffer() 将失败，结果代码为 -12743，“表示给定媒体的格式与给定的格式描述不匹配。例如，格式描述与 CMVideoFormatDescriptionMatchesImageBuffer 失败的 CVImageBuffer 配对。"

你可以看到 Kametrixom 和我都使用 CMSampleBufferGetFormatDescription(sampleBuffer) 来尝试复制源缓冲区的格式描述。因此，我不确定为什么给定媒体的格式与给定的格式描述不匹配。

Answer 1

好吧，我想我终于明白了。我创建了一个辅助扩展来制作 CVPixelBuffer:

的完整副本

extension CVPixelBuffer {
    func copy() -> CVPixelBuffer {
        precondition(CFGetTypeID(self) == CVPixelBufferGetTypeID(), "copy() cannot be called on a non-CVPixelBuffer")

        var _copy : CVPixelBuffer?
        CVPixelBufferCreate(
            nil,
            CVPixelBufferGetWidth(self),
            CVPixelBufferGetHeight(self),
            CVPixelBufferGetPixelFormatType(self),
            CVBufferGetAttachments(self, kCVAttachmentMode_ShouldPropagate)?.takeUnretainedValue(),
            &_copy)

        guard let copy = _copy else { fatalError() }

        CVPixelBufferLockBaseAddress(self, kCVPixelBufferLock_ReadOnly)
        CVPixelBufferLockBaseAddress(copy, 0)

        for plane in 0..<CVPixelBufferGetPlaneCount(self) {
            let dest = CVPixelBufferGetBaseAddressOfPlane(copy, plane)
            let source = CVPixelBufferGetBaseAddressOfPlane(self, plane)
            let height = CVPixelBufferGetHeightOfPlane(self, plane)
            let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(self, plane)

            memcpy(dest, source, height * bytesPerRow)
        }

        CVPixelBufferUnlockBaseAddress(copy, 0)
        CVPixelBufferUnlockBaseAddress(self, kCVPixelBufferLock_ReadOnly)

        return copy
    }
}

现在您可以在 didOutputSampleBuffer 方法中使用它：

guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }

let copy = pixelBuffer.copy()

toProcess.append(copy)

但请注意，一个这样的 pixelBuffer 占用大约 3MB 的内存 (1080p)，这意味着在 100 帧中您已经获得了大约 300MB，这大约是 iPhone 表示 STAHP（和崩溃）。

请注意，您实际上并不想复制 CMSampleBuffer，因为它实际上只包含一个 CVPixelBuffer，因为它是一个图像。

Answer 2

这是评分最高答案的 Swift 3 解决方案。

extension CVPixelBuffer {
func copy() -> CVPixelBuffer {
    precondition(CFGetTypeID(self) == CVPixelBufferGetTypeID(), "copy() cannot be called on a non-CVPixelBuffer")

    var _copy : CVPixelBuffer?
    CVPixelBufferCreate(
        kCFAllocatorDefault,
        CVPixelBufferGetWidth(self),
        CVPixelBufferGetHeight(self),
        CVPixelBufferGetPixelFormatType(self),
        nil,
        &_copy)

    guard let copy = _copy else { fatalError() }

    CVPixelBufferLockBaseAddress(self, CVPixelBufferLockFlags.readOnly)
    CVPixelBufferLockBaseAddress(copy, CVPixelBufferLockFlags(rawValue: 0))


    let copyBaseAddress = CVPixelBufferGetBaseAddress(copy)
    let currBaseAddress = CVPixelBufferGetBaseAddress(self)

    memcpy(copyBaseAddress, currBaseAddress, CVPixelBufferGetDataSize(self))

    CVPixelBufferUnlockBaseAddress(copy, CVPixelBufferLockFlags(rawValue: 0))
    CVPixelBufferUnlockBaseAddress(self, CVPixelBufferLockFlags.readOnly)


    return copy
}
}

Answer 3

我相信通过 VideoToolbox.framework，您可以使用 VTPixelTransferSession 来复制像素缓冲区。事实上，这是 class 做的唯一一件事。

参考： https://developer.apple.com/documentation/videotoolbox/vtpixeltransfersession-7cg

Answer 4

我花了好几个小时试图让它工作。事实证明，原始 CVPixelBuffer 的附件和 PixelBufferMetalCompatibilityKey 中的 IOSurface 选项都是必需的。

（仅供参考，遗憾的是，VideoToolbox 的 PixelTransferSession 仅适用于 macOS。）

这是我最后得到的。我在最后留下了几行，允许您通过比较原始和复制的 CVPixelBuffers 的平均颜色来验证 memcpy。它确实会减慢速度，所以一旦您确信您的 copy() 正在按预期工作，就应该将其删除。 CIImage.averageColour 扩展改编自 this code。

extension CVPixelBuffer {
    func copy() -> CVPixelBuffer {
        precondition(CFGetTypeID(self) == CVPixelBufferGetTypeID(), "copy() cannot be called on a non-CVPixelBuffer")

        let ioSurfaceProps = [
            "IOSurfaceOpenGLESFBOCompatibility": true as CFBoolean,
            "IOSurfaceOpenGLESTextureCompatibility": true as CFBoolean,
            "IOSurfaceCoreAnimationCompatibility": true as CFBoolean
        ] as CFDictionary

        let options = [
            String(kCVPixelBufferMetalCompatibilityKey): true as CFBoolean,
            String(kCVPixelBufferIOSurfacePropertiesKey): ioSurfaceProps
        ] as CFDictionary

        var _copy : CVPixelBuffer?
        CVPixelBufferCreate(
            nil,
            CVPixelBufferGetWidth(self),
            CVPixelBufferGetHeight(self),
            CVPixelBufferGetPixelFormatType(self),
            options,
            &_copy)

        guard let copy = _copy else { fatalError() }

        CVBufferPropagateAttachments(self as CVBuffer, copy as CVBuffer)

        CVPixelBufferLockBaseAddress(self, CVPixelBufferLockFlags.readOnly)
        CVPixelBufferLockBaseAddress(copy, CVPixelBufferLockFlags(rawValue: 0))

        let copyBaseAddress = CVPixelBufferGetBaseAddress(copy)
        let currBaseAddress = CVPixelBufferGetBaseAddress(self)

        memcpy(copyBaseAddress, currBaseAddress, CVPixelBufferGetDataSize(self))

        CVPixelBufferUnlockBaseAddress(copy, CVPixelBufferLockFlags(rawValue: 0))
        CVPixelBufferUnlockBaseAddress(self, CVPixelBufferLockFlags.readOnly)

        // let's make sure they have the same average color
//        let originalImage = CIImage(cvPixelBuffer: self)
//        let copiedImage = CIImage(cvPixelBuffer: copy)
//
//        let averageColorOriginal = originalImage.averageColour()
//        let averageColorCopy = copiedImage.averageColour()
//
//        assert(averageColorCopy == averageColorOriginal)
//        debugPrint("average frame color: \(averageColorCopy)")

        return copy
    }
}

从 CMSampleBuffer 中提取数据以创建深拷贝

Pulling data from a CMSampleBuffer in order to create a deep copy

ios

swift

deep-copy

pool

cmsamplebuffer