保存来自 TrueDepth 相机的深度图像

Question

我正在尝试保存来自 iPhoneX TrueDepth 相机的深度图像。使用 AVCamPhotoFilter 示例代码，我能够在 phone 的屏幕上实时查看转换为灰度格式的深度。我不知道如何以原始（16 位或更多位）格式保存深度图像序列。

我有 depthData，它是 AVDepthData 的一个实例。它的成员之一是 depthDataMap，它是 CVPixelBuffer 和图像格式类型 kCVPixelFormatType_DisparityFloat16 的实例。有没有办法把它保存到phone以供离线操作传输？

Answer 1

"raw" depth/disparity 地图没有标准的视频格式，这可能与 AVCapture 没有真正提供记录方式有关。

您有几个值得研究的选项：

将深度图转换为灰度纹理（您可以使用 AVCamPhotoFilter 示例代码中的代码来完成），然后将这些纹理传递给 AVAssetWriter 以生成灰度视频.根据您选择的视频格式和灰度转换方法，您为阅读视频而编写的其他软件可能能够从灰度帧中恢复 depth/disparity 足够精确的信息以满足您的目的。
只要你有一个 CVPixelBuffer，你就可以自己获取数据并用它做任何你想做的事。使用 CVPixelBufferLockBaseAddress (with the readOnly flag) to make sure the content won't change while you read it, then copy data from the pointer CVPixelBufferGetBaseAddress 提供给任何你想要的地方。（使用其他像素缓冲区函数查看要复制多少字节，完成后解锁缓冲区。）

但请注意：如果您花费太多时间从缓冲区复制或以其他方式保留它们，它们将不会在新缓冲区从捕获系统进入时被释放，并且您的捕获会话将挂起。（总而言之，如果不测试设备是否具有内存和 I/O 带宽以这种方式进行大量记录，则不清楚。）

Answer 2

您可以使用压缩库来创建包含原始 CVPixelBuffer 数据的 zip 文件。这个解决方案几乎没有问题。

数据很多，zip 不是很好的压缩方式。（压缩文件比帧数相同的每帧 32 位视频大 20 倍）。
Apple 的压缩库创建了一个标准 zip 程序无法打开的文件。我在 C 代码中使用 zlib 来读取它并使用 inflateInit2(&strm, -15); 使其工作。
您需要做一些工作才能将文件从您的应用程序中导出

这是我的代码（我将其限制为 250 帧，因为它保存在 RAM 中，但如果需要更多帧，您可以刷新到磁盘）：

//  DepthCapture.swift
//  AVCamPhotoFilter
//
//  Created by Eyal Fink on 07/04/2018.
//  Copyright © 2018 Resonai. All rights reserved.
//
// Capture the depth pixelBuffer into a compress file.
// This is very hacky and there are lots of TODOs but instead we need to replace
// it with a much better compression (video compression)....

import AVFoundation
import Foundation
import Compression


class DepthCapture {
    let kErrorDomain = "DepthCapture"
    let maxNumberOfFrame = 250
    lazy var bufferSize = 640 * 480 * 2 * maxNumberOfFrame  // maxNumberOfFrame frames
    var dstBuffer: UnsafeMutablePointer<UInt8>?
    var frameCount: Int64 = 0
    var outputURL: URL?
    var compresserPtr: UnsafeMutablePointer<compression_stream>?
    var file: FileHandle?

    // All operations handling the compresser oobjects are done on the
    // porcessingQ so they will happen sequentially
    var processingQ = DispatchQueue(label: "compression",
                                    qos: .userInteractive)


    func reset() {
        frameCount = 0
        outputURL = nil
        if self.compresserPtr != nil {
            //free(compresserPtr!.pointee.dst_ptr)
            compression_stream_destroy(self.compresserPtr!)
            self.compresserPtr = nil
        }
        if self.file != nil {
            self.file!.closeFile()
            self.file = nil
        }
    }
    func prepareForRecording() {
        reset()
        // Create the output zip file, remove old one if exists
        let documentsPath = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0] as NSString
        self.outputURL = URL(fileURLWithPath: documentsPath.appendingPathComponent("Depth"))
        FileManager.default.createFile(atPath: self.outputURL!.path, contents: nil, attributes: nil)
        self.file = FileHandle(forUpdatingAtPath: self.outputURL!.path)
        if self.file == nil {
            NSLog("Cannot create file at: \(self.outputURL!.path)")
            return
        }

        // Init the compression object
        compresserPtr = UnsafeMutablePointer<compression_stream>.allocate(capacity: 1)
        compression_stream_init(compresserPtr!, COMPRESSION_STREAM_ENCODE, COMPRESSION_ZLIB)
        dstBuffer = UnsafeMutablePointer<UInt8>.allocate(capacity: bufferSize)
        compresserPtr!.pointee.dst_ptr = dstBuffer!
        //defer { free(bufferPtr) }
        compresserPtr!.pointee.dst_size = bufferSize


    }
    func flush() {
        //let data = Data(bytesNoCopy: compresserPtr!.pointee.dst_ptr, count: bufferSize, deallocator: .none)
        let nBytes = bufferSize - compresserPtr!.pointee.dst_size
        print("Writing \(nBytes)")
        let data = Data(bytesNoCopy: dstBuffer!, count: nBytes, deallocator: .none)
        self.file?.write(data)
    }

    func startRecording() throws {
        processingQ.async {
            self.prepareForRecording()
        }
    }
    func addPixelBuffers(pixelBuffer: CVPixelBuffer) {
        processingQ.async {
            if self.frameCount >= self.maxNumberOfFrame {
                // TODO now!! flush when needed!!!
                print("MAXED OUT")
                return
            }

            CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly)
            let add : UnsafeMutableRawPointer = CVPixelBufferGetBaseAddress(pixelBuffer)!
            self.compresserPtr!.pointee.src_ptr = UnsafePointer<UInt8>(add.assumingMemoryBound(to: UInt8.self))
            let height = CVPixelBufferGetHeight(pixelBuffer)
            self.compresserPtr!.pointee.src_size = CVPixelBufferGetBytesPerRow(pixelBuffer) * height
            let flags = Int32(0)
            let compression_status = compression_stream_process(self.compresserPtr!, flags)
            if compression_status != COMPRESSION_STATUS_OK {
                NSLog("Buffer compression retured: \(compression_status)")
                return
            }
            if self.compresserPtr!.pointee.src_size != 0 {
                NSLog("Compression lib didn't eat all data: \(compression_status)")
                return
            }
            CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly)
            // TODO(eyal): flush when needed!!!
            self.frameCount += 1
            print("handled \(self.frameCount) buffers")
        }
    }
    func finishRecording(success: @escaping ((URL) -> Void)) throws {
        processingQ.async {
            let flags = Int32(COMPRESSION_STREAM_FINALIZE.rawValue)
            self.compresserPtr!.pointee.src_size = 0
            //compresserPtr!.pointee.src_ptr = UnsafePointer<UInt8>(0)
            let compression_status = compression_stream_process(self.compresserPtr!, flags)
            if compression_status != COMPRESSION_STATUS_END {
                NSLog("ERROR: Finish failed. compression retured: \(compression_status)")
                return
            }
            self.flush()
            DispatchQueue.main.sync {
                success(self.outputURL!)
            }
            self.reset()
        }
    }
}

保存来自 TrueDepth 相机的深度图像

Save depth images from TrueDepth camera

ios

swift

iphone-x

truedepth-camera