从 CMBlockBuffer 中提取 h264
Extracting h264 from CMBlockBuffer
我正在使用 Apple VideoTool Box (iOS) 压缩设备相机捕获的原始帧。
正在使用包含 CMBlockBuffer 的 CMSampleBufferRef 对象调用我的回调。
CMBlockBuffer 对象包含 H264 基本流，但我没有找到任何方法来获取指向基本流的指针。
当我将 CMSampleBufferRef 对象打印到控制台时，我得到：
(lldb) po blockBufferRef
CMBlockBuffer 0x1701193e0 totalDataLength: 4264 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
[0] 4264 bytes @ offset 128 Buffer Reference:
CMBlockBuffer 0x170119350 totalDataLength: 4632 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
[0] 4632 bytes @ offset 0 Memory Block 0x10295c000, 4632 bytes (custom V=0 A=0x0 F=0x18498bb44 R=0x0)
我设法获取指针的 CMBlockBuffer 对象似乎包含另一个不可访问的 CMBlockBuferRef（4632 字节）。
任何人都可以 post 如何访问 H264 elemantry 流？
谢谢！
我自己也为此苦苦挣扎了一段时间，终于弄明白了。
函数 CMBlockBufferGetDataPointer 可让您访问所需的所有数据，但要将其转换为基本流，您需要做一些不太明显的事情。
AVCC 与附件 B 格式
CMBlockBuffer 中的数据以 AVCC 格式存储，而基本流通常遵循 Annex B 规范（here 是对这两种格式的出色概述）。在 AVCC 格式中，前 4 个字节包含 NAL 单元的长度（H264 数据包的另一种说法）。您需要将此 header 替换为 4 字节起始代码：0x00 0x00 0x00 0x01，它在 Annex B 基本流中用作 NAL 单元之间的分隔符（3 字节版本 0x00 0x00 0x01 也可以正常工作）。
单个 CMBlockBuffer 中的多个 NAL 单元
下一个不是很明显的事情是单个 CMBlockBuffer 有时会包含多个 NAL 单元。 Apple 似乎向每个 I-Frame NAL 单元（也称为 IDR）添加了一个包含元数据的额外 NAL 单元 (SEI)。这可能就是您在单个 CMBlockBuffer object 中看到多个缓冲区的原因。但是，CMBlockBufferGetDataPointer 函数为您提供了一个可以访问所有数据的指针。也就是说，多个 NAL 单元的存在使 AVCC header 的转换复杂化。现在你实际上必须读取 AVCC header 中包含的长度值以找到下一个 NAL 单元，并继续转换 headers 直到到达缓冲区的末尾。
Big-Endian 对比 Little-Endian
接下来不是很明显的事情是AVCC header 以Big-Endian 格式存储，而iOS 是Little-Endian 本机。因此，当您读取 AVCC header 中包含的长度值时，首先将其传递给 CFSwapInt32BigToHost 函数。
SPS 和 PPS NAL 单元
最后不是很明显的是CMBlockBuffer里面的数据不包含参数NAL units SPS和PPS，里面包含解码器的配置参数，比如profile，level，resolution，frame rate。这些作为元数据存储在样本缓冲区的格式描述中，可以通过函数 CMVideoFormatDescriptionGetH264ParameterSetAtIndex 访问。请注意，您必须在发送前将起始代码添加到这些 NAL 单元。 SPS 和 PPS NAL 单元不必与每个新帧一起发送。解码器只需要读取一次，但通常会定期重新发送它们，例如在每个新的 I-frame NAL 单元之前。
代码示例
下面是一个考虑了所有这些因素的代码示例。
static void videoFrameFinishedEncoding(void *outputCallbackRefCon,
void *sourceFrameRefCon,
OSStatus status,
VTEncodeInfoFlags infoFlags,
CMSampleBufferRef sampleBuffer) {
// Check if there were any errors encoding
if (status != noErr) {
NSLog(@"Error encoding video, err=%lld", (int64_t)status);
return;
}
// In this example we will use a NSMutableData object to store the
// elementary stream.
NSMutableData *elementaryStream = [NSMutableData data];
// Find out if the sample buffer contains an I-Frame.
// If so we will write the SPS and PPS NAL units to the elementary stream.
BOOL isIFrame = NO;
CFArrayRef attachmentsArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0);
if (CFArrayGetCount(attachmentsArray)) {
CFBooleanRef notSync;
CFDictionaryRef dict = CFArrayGetValueAtIndex(attachmentsArray, 0);
BOOL keyExists = CFDictionaryGetValueIfPresent(dict,
kCMSampleAttachmentKey_NotSync,
(const void **)¬Sync);
// An I-Frame is a sync frame
isIFrame = !keyExists || !CFBooleanGetValue(notSync);
}
// This is the start code that we will write to
// the elementary stream before every NAL unit
static const size_t startCodeLength = 4;
static const uint8_t startCode[] = {0x00, 0x00, 0x00, 0x01};
// Write the SPS and PPS NAL units to the elementary stream before every I-Frame
if (isIFrame) {
CMFormatDescriptionRef description = CMSampleBufferGetFormatDescription(sampleBuffer);
// Find out how many parameter sets there are
size_t numberOfParameterSets;
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
0, NULL, NULL,
&numberOfParameterSets,
NULL);
// Write each parameter set to the elementary stream
for (int i = 0; i < numberOfParameterSets; i++) {
const uint8_t *parameterSetPointer;
size_t parameterSetLength;
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
i,
¶meterSetPointer,
¶meterSetLength,
NULL, NULL);
// Write the parameter set to the elementary stream
[elementaryStream appendBytes:startCode length:startCodeLength];
[elementaryStream appendBytes:parameterSetPointer length:parameterSetLength];
}
}
// Get a pointer to the raw AVCC NAL unit data in the sample buffer
size_t blockBufferLength;
uint8_t *bufferDataPointer = NULL;
CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(sampleBuffer),
0,
NULL,
&blockBufferLength,
(char **)&bufferDataPointer);
// Loop through all the NAL units in the block buffer
// and write them to the elementary stream with
// start codes instead of AVCC length headers
size_t bufferOffset = 0;
static const int AVCCHeaderLength = 4;
while (bufferOffset < blockBufferLength - AVCCHeaderLength) {
// Read the NAL unit length
uint32_t NALUnitLength = 0;
memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength);
// Convert the length value from Big-endian to Little-endian
NALUnitLength = CFSwapInt32BigToHost(NALUnitLength);
// Write start code to the elementary stream
[elementaryStream appendBytes:startCode length:startCodeLength];
// Write the NAL unit without the AVCC length header to the elementary stream
[elementaryStream appendBytes:bufferDataPointer + bufferOffset + AVCCHeaderLength
length:NALUnitLength];
// Move to the next NAL unit in the block buffer
bufferOffset += AVCCHeaderLength + NALUnitLength;
}
}
感谢安东的出色回答！我正在为有兴趣直接在基于 Swift 的项目中使用此处讨论的概念的人们提供您的解决方案的简单 Swift 端口。
public func didEncodeFrame(frame: CMSampleBuffer)
{
print ("Received encoded frame in delegate...")
//----AVCC to Elem stream-----//
var elementaryStream = NSMutableData()
//1. check if CMBuffer had I-frame
var isIFrame:Bool = false
let attachmentsArray:CFArray = CMSampleBufferGetSampleAttachmentsArray(frame, false)!
//check how many attachments
if ( CFArrayGetCount(attachmentsArray) > 0 ) {
let dict = CFArrayGetValueAtIndex(attachmentsArray, 0)
let dictRef:CFDictionaryRef = unsafeBitCast(dict, CFDictionaryRef.self)
//get value
let value = CFDictionaryGetValue(dictRef, unsafeBitCast(kCMSampleAttachmentKey_NotSync, UnsafePointer<Void>.self))
if ( value != nil ){
print ("IFrame found...")
isIFrame = true
}
}
//2. define the start code
let nStartCodeLength:size_t = 4
let nStartCode:[UInt8] = [0x00, 0x00, 0x00, 0x01]
//3. write the SPS and PPS before I-frame
if ( isIFrame == true ){
let description:CMFormatDescriptionRef = CMSampleBufferGetFormatDescription(frame)!
//how many params
var numParams:size_t = 0
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, 0, nil, nil, &numParams, nil)
//write each param-set to elementary stream
print("Write param to elementaryStream ", numParams)
for i in 0..<numParams {
var parameterSetPointer:UnsafePointer<UInt8> = nil
var parameterSetLength:size_t = 0
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, i, ¶meterSetPointer, ¶meterSetLength, nil, nil)
elementaryStream.appendBytes(nStartCode, length: nStartCodeLength)
elementaryStream.appendBytes(parameterSetPointer, length: unsafeBitCast(parameterSetLength, Int.self))
}
}
//4. Get a pointer to the raw AVCC NAL unit data in the sample buffer
var blockBufferLength:size_t = 0
var bufferDataPointer: UnsafeMutablePointer<Int8> = nil
CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(frame)!, 0, nil, &blockBufferLength, &bufferDataPointer)
print ("Block length = ", blockBufferLength)
//5. Loop through all the NAL units in the block buffer
var bufferOffset:size_t = 0
let AVCCHeaderLength:Int = 4
while (bufferOffset < (blockBufferLength - AVCCHeaderLength) ) {
// Read the NAL unit length
var NALUnitLength:UInt32 = 0
memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength)
//Big-Endian to Little-Endian
NALUnitLength = CFSwapInt32(NALUnitLength)
if ( NALUnitLength > 0 ){
print ( "NALUnitLen = ", NALUnitLength)
// Write start code to the elementary stream
elementaryStream.appendBytes(nStartCode, length: nStartCodeLength)
// Write the NAL unit without the AVCC length header to the elementary stream
elementaryStream.appendBytes(bufferDataPointer + bufferOffset + AVCCHeaderLength, length: Int(NALUnitLength))
// Move to the next NAL unit in the block buffer
bufferOffset += AVCCHeaderLength + size_t(NALUnitLength);
print("Moving to next NALU...")
}
}
print("Read completed...")
}
我正在使用 Apple VideoTool Box (iOS) 压缩设备相机捕获的原始帧。
正在使用包含 CMBlockBuffer 的 CMSampleBufferRef 对象调用我的回调。
CMBlockBuffer 对象包含 H264 基本流，但我没有找到任何方法来获取指向基本流的指针。
当我将 CMSampleBufferRef 对象打印到控制台时，我得到：
(lldb) po blockBufferRef
CMBlockBuffer 0x1701193e0 totalDataLength: 4264 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
[0] 4264 bytes @ offset 128 Buffer Reference:
CMBlockBuffer 0x170119350 totalDataLength: 4632 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
[0] 4632 bytes @ offset 0 Memory Block 0x10295c000, 4632 bytes (custom V=0 A=0x0 F=0x18498bb44 R=0x0)
我设法获取指针的 CMBlockBuffer 对象似乎包含另一个不可访问的 CMBlockBuferRef（4632 字节）。
任何人都可以 post 如何访问 H264 elemantry 流？
谢谢！
我自己也为此苦苦挣扎了一段时间，终于弄明白了。
函数 CMBlockBufferGetDataPointer 可让您访问所需的所有数据，但要将其转换为基本流，您需要做一些不太明显的事情。
AVCC 与附件 B 格式
CMBlockBuffer 中的数据以 AVCC 格式存储，而基本流通常遵循 Annex B 规范（here 是对这两种格式的出色概述）。在 AVCC 格式中，前 4 个字节包含 NAL 单元的长度（H264 数据包的另一种说法）。您需要将此 header 替换为 4 字节起始代码：0x00 0x00 0x00 0x01，它在 Annex B 基本流中用作 NAL 单元之间的分隔符（3 字节版本 0x00 0x00 0x01 也可以正常工作）。
单个 CMBlockBuffer 中的多个 NAL 单元
下一个不是很明显的事情是单个 CMBlockBuffer 有时会包含多个 NAL 单元。 Apple 似乎向每个 I-Frame NAL 单元（也称为 IDR）添加了一个包含元数据的额外 NAL 单元 (SEI)。这可能就是您在单个 CMBlockBuffer object 中看到多个缓冲区的原因。但是，CMBlockBufferGetDataPointer 函数为您提供了一个可以访问所有数据的指针。也就是说，多个 NAL 单元的存在使 AVCC header 的转换复杂化。现在你实际上必须读取 AVCC header 中包含的长度值以找到下一个 NAL 单元，并继续转换 headers 直到到达缓冲区的末尾。
Big-Endian 对比 Little-Endian
接下来不是很明显的事情是AVCC header 以Big-Endian 格式存储，而iOS 是Little-Endian 本机。因此，当您读取 AVCC header 中包含的长度值时，首先将其传递给 CFSwapInt32BigToHost 函数。
SPS 和 PPS NAL 单元
最后不是很明显的是CMBlockBuffer里面的数据不包含参数NAL units SPS和PPS，里面包含解码器的配置参数，比如profile，level，resolution，frame rate。这些作为元数据存储在样本缓冲区的格式描述中，可以通过函数 CMVideoFormatDescriptionGetH264ParameterSetAtIndex 访问。请注意，您必须在发送前将起始代码添加到这些 NAL 单元。 SPS 和 PPS NAL 单元不必与每个新帧一起发送。解码器只需要读取一次，但通常会定期重新发送它们，例如在每个新的 I-frame NAL 单元之前。
代码示例
下面是一个考虑了所有这些因素的代码示例。
static void videoFrameFinishedEncoding(void *outputCallbackRefCon,
void *sourceFrameRefCon,
OSStatus status,
VTEncodeInfoFlags infoFlags,
CMSampleBufferRef sampleBuffer) {
// Check if there were any errors encoding
if (status != noErr) {
NSLog(@"Error encoding video, err=%lld", (int64_t)status);
return;
}
// In this example we will use a NSMutableData object to store the
// elementary stream.
NSMutableData *elementaryStream = [NSMutableData data];
// Find out if the sample buffer contains an I-Frame.
// If so we will write the SPS and PPS NAL units to the elementary stream.
BOOL isIFrame = NO;
CFArrayRef attachmentsArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0);
if (CFArrayGetCount(attachmentsArray)) {
CFBooleanRef notSync;
CFDictionaryRef dict = CFArrayGetValueAtIndex(attachmentsArray, 0);
BOOL keyExists = CFDictionaryGetValueIfPresent(dict,
kCMSampleAttachmentKey_NotSync,
(const void **)¬Sync);
// An I-Frame is a sync frame
isIFrame = !keyExists || !CFBooleanGetValue(notSync);
}
// This is the start code that we will write to
// the elementary stream before every NAL unit
static const size_t startCodeLength = 4;
static const uint8_t startCode[] = {0x00, 0x00, 0x00, 0x01};
// Write the SPS and PPS NAL units to the elementary stream before every I-Frame
if (isIFrame) {
CMFormatDescriptionRef description = CMSampleBufferGetFormatDescription(sampleBuffer);
// Find out how many parameter sets there are
size_t numberOfParameterSets;
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
0, NULL, NULL,
&numberOfParameterSets,
NULL);
// Write each parameter set to the elementary stream
for (int i = 0; i < numberOfParameterSets; i++) {
const uint8_t *parameterSetPointer;
size_t parameterSetLength;
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
i,
¶meterSetPointer,
¶meterSetLength,
NULL, NULL);
// Write the parameter set to the elementary stream
[elementaryStream appendBytes:startCode length:startCodeLength];
[elementaryStream appendBytes:parameterSetPointer length:parameterSetLength];
}
}
// Get a pointer to the raw AVCC NAL unit data in the sample buffer
size_t blockBufferLength;
uint8_t *bufferDataPointer = NULL;
CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(sampleBuffer),
0,
NULL,
&blockBufferLength,
(char **)&bufferDataPointer);
// Loop through all the NAL units in the block buffer
// and write them to the elementary stream with
// start codes instead of AVCC length headers
size_t bufferOffset = 0;
static const int AVCCHeaderLength = 4;
while (bufferOffset < blockBufferLength - AVCCHeaderLength) {
// Read the NAL unit length
uint32_t NALUnitLength = 0;
memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength);
// Convert the length value from Big-endian to Little-endian
NALUnitLength = CFSwapInt32BigToHost(NALUnitLength);
// Write start code to the elementary stream
[elementaryStream appendBytes:startCode length:startCodeLength];
// Write the NAL unit without the AVCC length header to the elementary stream
[elementaryStream appendBytes:bufferDataPointer + bufferOffset + AVCCHeaderLength
length:NALUnitLength];
// Move to the next NAL unit in the block buffer
bufferOffset += AVCCHeaderLength + NALUnitLength;
}
}
感谢安东的出色回答！我正在为有兴趣直接在基于 Swift 的项目中使用此处讨论的概念的人们提供您的解决方案的简单 Swift 端口。
public func didEncodeFrame(frame: CMSampleBuffer)
{
print ("Received encoded frame in delegate...")
//----AVCC to Elem stream-----//
var elementaryStream = NSMutableData()
//1. check if CMBuffer had I-frame
var isIFrame:Bool = false
let attachmentsArray:CFArray = CMSampleBufferGetSampleAttachmentsArray(frame, false)!
//check how many attachments
if ( CFArrayGetCount(attachmentsArray) > 0 ) {
let dict = CFArrayGetValueAtIndex(attachmentsArray, 0)
let dictRef:CFDictionaryRef = unsafeBitCast(dict, CFDictionaryRef.self)
//get value
let value = CFDictionaryGetValue(dictRef, unsafeBitCast(kCMSampleAttachmentKey_NotSync, UnsafePointer<Void>.self))
if ( value != nil ){
print ("IFrame found...")
isIFrame = true
}
}
//2. define the start code
let nStartCodeLength:size_t = 4
let nStartCode:[UInt8] = [0x00, 0x00, 0x00, 0x01]
//3. write the SPS and PPS before I-frame
if ( isIFrame == true ){
let description:CMFormatDescriptionRef = CMSampleBufferGetFormatDescription(frame)!
//how many params
var numParams:size_t = 0
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, 0, nil, nil, &numParams, nil)
//write each param-set to elementary stream
print("Write param to elementaryStream ", numParams)
for i in 0..<numParams {
var parameterSetPointer:UnsafePointer<UInt8> = nil
var parameterSetLength:size_t = 0
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, i, ¶meterSetPointer, ¶meterSetLength, nil, nil)
elementaryStream.appendBytes(nStartCode, length: nStartCodeLength)
elementaryStream.appendBytes(parameterSetPointer, length: unsafeBitCast(parameterSetLength, Int.self))
}
}
//4. Get a pointer to the raw AVCC NAL unit data in the sample buffer
var blockBufferLength:size_t = 0
var bufferDataPointer: UnsafeMutablePointer<Int8> = nil
CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(frame)!, 0, nil, &blockBufferLength, &bufferDataPointer)
print ("Block length = ", blockBufferLength)
//5. Loop through all the NAL units in the block buffer
var bufferOffset:size_t = 0
let AVCCHeaderLength:Int = 4
while (bufferOffset < (blockBufferLength - AVCCHeaderLength) ) {
// Read the NAL unit length
var NALUnitLength:UInt32 = 0
memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength)
//Big-Endian to Little-Endian
NALUnitLength = CFSwapInt32(NALUnitLength)
if ( NALUnitLength > 0 ){
print ( "NALUnitLen = ", NALUnitLength)
// Write start code to the elementary stream
elementaryStream.appendBytes(nStartCode, length: nStartCodeLength)
// Write the NAL unit without the AVCC length header to the elementary stream
elementaryStream.appendBytes(bufferDataPointer + bufferOffset + AVCCHeaderLength, length: Int(NALUnitLength))
// Move to the next NAL unit in the block buffer
bufferOffset += AVCCHeaderLength + size_t(NALUnitLength);
print("Moving to next NALU...")
}
}
print("Read completed...")
}