如何为 YCbCr 图像创建 vImage_CGImageFormat？
How to create vImage_CGImageFormat for YCbCr image?
我想创建一个库，可以将使用 AVCaptureDevice 捕获的任何视频帧转换为目标像素格式。因此，它必须支持 kCVPixelFormatType_32BGRA、kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange、kCVPixelFormatType_420YpCbCr8BiPlanarFullRange 和可选的 kCVPixelFormatType_420YpCbCr8Planar.
为了提高性能，我想使用 Accelerate 框架，因为它包含丰富的 conversion functions. As target pixel format may be different and is set by library user it'll be grate to use universal vImageConvert_AnyToAny 功能集：
You use vImage's vImageConvert_AnyToAny(::::_:) function to
convert between Core Video or Core Graphics image data of arbitrary
color spaces and bit depths. The source and destination images are
described by one or more buffers. For example, a Y'CbCr image may be
composed of one buffer containing luminance information and one buffer
containing chrominance information.
要使用此函数，我必须创建 vImageConverter which define conversion between images. Constructors of this class require source and destination image format description in form of vImage_CGImageFormat:
vImage_CGImageFormat in_format = ...;
vImage_CGImageFormat out_format = ...;
vImageConverterRef converter = vImageConverter_CreateWithCGImageFormat(&in_format, &out_format, NULL, kvImagePrintDiagnosticsToConsole, &err);
if( err == kvImageNoError )
{
vImage_Buffer *src_planes = ...;
vImage_Buffer *dst_planes = ...;
err = vImageConvert_AnyToAny(converter, src_planes, dst_planes, NULL, kvImagePrintDiagnosticsToConsole);
}
代码基于这篇 Apple 文章：Building a Basic Conversion Workflow
对于kCVPixelFormatType_32BGRA这样的vImage_CGImageFormat很简单，在vImage_Utilities.h:
中有描述
/*!
* @struct vImage_CGImageFormat
* @abstract A pixel format
* @discussion A vImage_CGImageFormat describes the ordering of the color channels, how many there are,
* the size and type of the data in the color channels and whether the data is premultiplied by alpha or not.
* This format mirrors the image format descriptors used by CoreGraphics to create things like CGImageRef and
* CGBitmapContextRef.
*
* This vImage_CGImageFormat:
*
* <pre>@textblock
* vImage_CGImageFormat format = {
* .bitsPerComponent = 8,
* .bitsPerPixel = 32,
* .colorSpace = CGColorSpaceCreateDeviceRGB(), // don't forget to release this!
* .bitmapInfo = kCGImageAlphaPremultipliedFirst | kCGBitmapByteOrder32Little,
* .version = 0, // must be 0
* .decode = NULL,
* .renderingIntent = kCGRenderingIntentDefault
* };
* @/textblock</pre>
*
* codes for a little endian ARGB8888 pixel, or what is called in the rest of vImage, BGRA8888. Note: for 16-
* and 32-bits per component formats (int16_t, uint16_t, half-float, float) most vImage image filters assume
* the data is in host-endian format. (The APIs in this header do not.) Host-endian is little endian for Intel
* and ARM, big endian for PowerPC. If the data is not in host-endian format, then you may use
* vImagePermuteChannels_ARGB8888 or vImageByteSwap_Planar16U to swap the image data byte ordering.
*
* Some examples:
* <pre>@textblock
* ARGB8888 -> {8, 32, NULL, alpha first, 0, NULL, kCGRenderingIntentDefault} alpha first = { kCGImageAlphaFirst, kCGImageAlphaPremultipliedFirst, kCGImageAlphaNoneSkipFirst }
* RGBA8888 -> {8, 32, NULL, alpha last, 0, NULL, kCGRenderingIntentDefault} alpha last = { kCGImageAlphaLast, kCGImageAlphaPremultipliedLast, kCGImageAlphaNoneSkipLast }
* BGRA8888 -> {8, 32, NULL, alpha first | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault}
* RGB888 -> {8, 24, NULL, kCGImageAlphaNone | kCGBitmapByteOrderDefault, 0, NULL, kCGRenderingIntentDefault}
* RGB565 -> {5, 16, NULL, kCGImageAlphaNone | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault}
* ARGB1555 -> {5, 16, NULL, alpha first | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault}
* RGBA16F -> {16, 64, NULL, alpha last | kCGBitmapFloatComponents | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault }
* CMYK8888 -> {8, 32, CGColorSpaceCreateDeviceCMYK(), kCGImageAlphaNone, 0, NULL, kCGRenderingIntentDefault }
* ARGBFFFF premultiplied -> {32, 128, NULL, kCGImageAlphaPremultipliedFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
* ARGBFFFF not-premultiplied -> {32, 128, NULL, kCGImageAlphaFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
* ARGBFFFF, alpha = 1 -> {32, 128, NULL, kCGImageAlphaNoneSkipFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
* @/textblock</pre>
*
* Note that some of these formats, particularly RGB565 and 16F formats are supported by vImage but
* not necessarily CoreGraphics. They will be converted to a higher precision format as necessary by
* vImage in vImageCreateCGImageFromBuffer().
*
* By C rules, uninitialized struct parameters are set to zero. The last three parameters are usually zero, so can usually be omitted.
*
* <pre>@textblock
* vImage_CGImageFormat srgb888 = (vImage_CGImageFormat){
* .bitsPerComponent = 8,
* .bitsPerPixel = 24,
* .colorSpace = NULL,
* .bitmapInfo = kCGImageAlphaNone | kCGBitmapByteOrderDefault };
* @/textblock</pre>
*
* To understand how these various parameters relate to one another, we can look at the process of converting from
* one vImage_CGImageFormat format to another:
*
* 1) transform endianness of src format given by bitmapInfo to host endian (except 8 bitPerComponent content)
* 2) remove decode array transformation, and up convert to a higher range format as necessary to preserve precision / range
* 3) convert src colorspace to reference XYZ colorspace (may cause upconvert to preserve range / precision)
* 4) convert XYZ to destination colorspace + rendering intent
* 5) convert to destination precision (given by bitsPerComponent)
* 6) deal with any alpha changes (given by bitmapInfo) or flattening that needs to occur
* 7) Apply any channel reordering requested, if it didn't happen at an earlier step. (As indicated by src and dest bitmapInfo)
* 8) Apply destination decode array
* 9) Apply endianness transform given by dest bitmapInfo
*
* Clearly, for most common transformations not all steps need to occur and multiple steps can be collapsed into a compound operation.
*
* @field bitsPerComponent The number of bits needed to represent one channel of data in one pixel. For ARGB8888, this would be 8. Expected values: {1, 2, 4, 5, 8, 10, 12, 16, 32}
* @field bitsPerPixel The number of bits needed to represent one pixel. For ARGB8888, this would be 32.
* It is possible that bitsPerPixel > bitsPerComponent * number of components, but in practice this is rare.
* The number of color components is given by the colorspace and the number of alpha components (0 or 1) is given by
* by the bitmapInfo.
* @field colorSpace A description of how the pixel data in the image is positioned relative to a reference XYZ color space.
* See CoreGraphics/CGColorSpace.h. Pass NULL as a shorthand for sRGB. The vImage_CGImageFormat is not
* capable of managing the memory held by the colorSpace. If you created the colorspace, you must
* be sure to release it before all references to it disappear from scope.
* @field bitmapInfo The CGBitmapInfo describing the color channels. See CoreGraphics/CGImage.h.
* ARGB8888 is kCGImageAlphaFirst | kCGBitmapByteOrderDefault
* BGRA8888 is kCGImageAlphaFirst | kCGBitmapByteOrder32Little
* @field version The struct is versioned for future expansion. Pass 0 here.
* @field decode Prior to transformations caused by the colorspace, color channels are subject to a linear transformation.
* This allows for a different range than the typical [0,1.0]. NULL indicates default behavior of [0,1.0]
* range, and is what you should use if you don't understand this parameter. See description of CGImageCreate()
* for a discussion of decode arrays. See also Decode Arrays section of Chapter 4.8 of the PDF specification.
* The vImage_CGImageFormat is not capable of managing the memory held by the decode array. If you created a
* decode array on the heap, you must be sure to release it before all references to it disappear from scope.
*
* @field renderingIntent See CGColorSpace.h. kCGRenderingIntentDefault is typical here. By convention, rendering intent changes that
* are not accompanied by a colorspace change are ignored.
*/
我不明白如何为 YCbCr 像素格式创建 vImage_CGImageFormat。
首先我认为它根本不受支持，但这种格式有特殊功能：Understanding YpCbCr Image Formats and vImageConverter has special functions for multi plane images such as vImageConverter_GetNumberOfSourceBuffers and vImageConverter_GetSourceBufferOrder. In the last function there are many vImage Buffer Type Codes甚至kvImageBufferTypeCode_CVPixelBuffer_YCbCr。
所以，看起来可以为 YCbCr 创建 vImageConverter，我需要帮助来了解如何做。
vImageConverter_CreateWithCGImageFormat 创建用于在两种 Core Graphics 格式之间进行转换的转换器。
在这种情况下，您不应该使用 vImageConverter_CreateForCVToCGImageFormat 将 CVPixelBuffer（带有可以使用 vImageCVImageFormat_CreateWithCVPixelBuffer 生成的 vImageCVImageFormat）转换为CGImage？ vImage_CGImageFormat 是您的目标格式 - 因此其属性由您定义。
有函数 vImageCreateRGBColorSpaceWithPrimariesAndTransferFunction 可以为 YCbCr 像素格式创建适当的颜色 space，但仅用于从 RGB 格式转换或转换为 RGB 格式：
const vImageTransferFunction f709 =
{
.c0 = 1.099,
.c1 = 1.0,
.c2 = 0.0,
.c3 = -0.099,
.gamma = 0.45,
.cutoff = 0.018,
.c4 = 4.5,
.c5 = 0
};
const vImageRGBPrimaries p709 =
{
.red_x = .64, // 1.3 red
.green_x = .30, // 1.3 green
.blue_x = .15, // 1.3 blue
.white_x = 0.3127, // 1.4 white
.red_y = .33, // 1.3 red
.green_y = .60, // 1.3 green
.blue_y = .06, // 1.3 blue
.white_y = 0.3290 // 1.4 white
};
vImage_Error err = kvImageNoError;
CGColorSpaceRef colorSpace = vImageCreateRGBColorSpaceWithPrimariesAndTransferFunction( &p709, &f709, kvImageNoFlags, &err );
vImage_CGImageFormat format = {
.bitsPerComponent = 8,
.bitsPerPixel = 32,
.bitmapInfo = (CGBitmapInfo)kCGImageAlphaNoneSkipFirst | kCGBitmapByteOrderDefault,
.colorSpace = colorSpace
};
我想创建一个库，可以将使用 AVCaptureDevice 捕获的任何视频帧转换为目标像素格式。因此，它必须支持 kCVPixelFormatType_32BGRA、kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange、kCVPixelFormatType_420YpCbCr8BiPlanarFullRange 和可选的 kCVPixelFormatType_420YpCbCr8Planar.
为了提高性能，我想使用 Accelerate 框架，因为它包含丰富的 conversion functions. As target pixel format may be different and is set by library user it'll be grate to use universal vImageConvert_AnyToAny 功能集：
You use vImage's vImageConvert_AnyToAny(::::_:) function to convert between Core Video or Core Graphics image data of arbitrary color spaces and bit depths. The source and destination images are described by one or more buffers. For example, a Y'CbCr image may be composed of one buffer containing luminance information and one buffer containing chrominance information.
要使用此函数，我必须创建 vImageConverter which define conversion between images. Constructors of this class require source and destination image format description in form of vImage_CGImageFormat:
vImage_CGImageFormat in_format = ...;
vImage_CGImageFormat out_format = ...;
vImageConverterRef converter = vImageConverter_CreateWithCGImageFormat(&in_format, &out_format, NULL, kvImagePrintDiagnosticsToConsole, &err);
if( err == kvImageNoError )
{
vImage_Buffer *src_planes = ...;
vImage_Buffer *dst_planes = ...;
err = vImageConvert_AnyToAny(converter, src_planes, dst_planes, NULL, kvImagePrintDiagnosticsToConsole);
}
代码基于这篇 Apple 文章：Building a Basic Conversion Workflow
对于kCVPixelFormatType_32BGRA这样的vImage_CGImageFormat很简单，在vImage_Utilities.h:
/*!
* @struct vImage_CGImageFormat
* @abstract A pixel format
* @discussion A vImage_CGImageFormat describes the ordering of the color channels, how many there are,
* the size and type of the data in the color channels and whether the data is premultiplied by alpha or not.
* This format mirrors the image format descriptors used by CoreGraphics to create things like CGImageRef and
* CGBitmapContextRef.
*
* This vImage_CGImageFormat:
*
* <pre>@textblock
* vImage_CGImageFormat format = {
* .bitsPerComponent = 8,
* .bitsPerPixel = 32,
* .colorSpace = CGColorSpaceCreateDeviceRGB(), // don't forget to release this!
* .bitmapInfo = kCGImageAlphaPremultipliedFirst | kCGBitmapByteOrder32Little,
* .version = 0, // must be 0
* .decode = NULL,
* .renderingIntent = kCGRenderingIntentDefault
* };
* @/textblock</pre>
*
* codes for a little endian ARGB8888 pixel, or what is called in the rest of vImage, BGRA8888. Note: for 16-
* and 32-bits per component formats (int16_t, uint16_t, half-float, float) most vImage image filters assume
* the data is in host-endian format. (The APIs in this header do not.) Host-endian is little endian for Intel
* and ARM, big endian for PowerPC. If the data is not in host-endian format, then you may use
* vImagePermuteChannels_ARGB8888 or vImageByteSwap_Planar16U to swap the image data byte ordering.
*
* Some examples:
* <pre>@textblock
* ARGB8888 -> {8, 32, NULL, alpha first, 0, NULL, kCGRenderingIntentDefault} alpha first = { kCGImageAlphaFirst, kCGImageAlphaPremultipliedFirst, kCGImageAlphaNoneSkipFirst }
* RGBA8888 -> {8, 32, NULL, alpha last, 0, NULL, kCGRenderingIntentDefault} alpha last = { kCGImageAlphaLast, kCGImageAlphaPremultipliedLast, kCGImageAlphaNoneSkipLast }
* BGRA8888 -> {8, 32, NULL, alpha first | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault}
* RGB888 -> {8, 24, NULL, kCGImageAlphaNone | kCGBitmapByteOrderDefault, 0, NULL, kCGRenderingIntentDefault}
* RGB565 -> {5, 16, NULL, kCGImageAlphaNone | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault}
* ARGB1555 -> {5, 16, NULL, alpha first | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault}
* RGBA16F -> {16, 64, NULL, alpha last | kCGBitmapFloatComponents | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault }
* CMYK8888 -> {8, 32, CGColorSpaceCreateDeviceCMYK(), kCGImageAlphaNone, 0, NULL, kCGRenderingIntentDefault }
* ARGBFFFF premultiplied -> {32, 128, NULL, kCGImageAlphaPremultipliedFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
* ARGBFFFF not-premultiplied -> {32, 128, NULL, kCGImageAlphaFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
* ARGBFFFF, alpha = 1 -> {32, 128, NULL, kCGImageAlphaNoneSkipFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
* @/textblock</pre>
*
* Note that some of these formats, particularly RGB565 and 16F formats are supported by vImage but
* not necessarily CoreGraphics. They will be converted to a higher precision format as necessary by
* vImage in vImageCreateCGImageFromBuffer().
*
* By C rules, uninitialized struct parameters are set to zero. The last three parameters are usually zero, so can usually be omitted.
*
* <pre>@textblock
* vImage_CGImageFormat srgb888 = (vImage_CGImageFormat){
* .bitsPerComponent = 8,
* .bitsPerPixel = 24,
* .colorSpace = NULL,
* .bitmapInfo = kCGImageAlphaNone | kCGBitmapByteOrderDefault };
* @/textblock</pre>
*
* To understand how these various parameters relate to one another, we can look at the process of converting from
* one vImage_CGImageFormat format to another:
*
* 1) transform endianness of src format given by bitmapInfo to host endian (except 8 bitPerComponent content)
* 2) remove decode array transformation, and up convert to a higher range format as necessary to preserve precision / range
* 3) convert src colorspace to reference XYZ colorspace (may cause upconvert to preserve range / precision)
* 4) convert XYZ to destination colorspace + rendering intent
* 5) convert to destination precision (given by bitsPerComponent)
* 6) deal with any alpha changes (given by bitmapInfo) or flattening that needs to occur
* 7) Apply any channel reordering requested, if it didn't happen at an earlier step. (As indicated by src and dest bitmapInfo)
* 8) Apply destination decode array
* 9) Apply endianness transform given by dest bitmapInfo
*
* Clearly, for most common transformations not all steps need to occur and multiple steps can be collapsed into a compound operation.
*
* @field bitsPerComponent The number of bits needed to represent one channel of data in one pixel. For ARGB8888, this would be 8. Expected values: {1, 2, 4, 5, 8, 10, 12, 16, 32}
* @field bitsPerPixel The number of bits needed to represent one pixel. For ARGB8888, this would be 32.
* It is possible that bitsPerPixel > bitsPerComponent * number of components, but in practice this is rare.
* The number of color components is given by the colorspace and the number of alpha components (0 or 1) is given by
* by the bitmapInfo.
* @field colorSpace A description of how the pixel data in the image is positioned relative to a reference XYZ color space.
* See CoreGraphics/CGColorSpace.h. Pass NULL as a shorthand for sRGB. The vImage_CGImageFormat is not
* capable of managing the memory held by the colorSpace. If you created the colorspace, you must
* be sure to release it before all references to it disappear from scope.
* @field bitmapInfo The CGBitmapInfo describing the color channels. See CoreGraphics/CGImage.h.
* ARGB8888 is kCGImageAlphaFirst | kCGBitmapByteOrderDefault
* BGRA8888 is kCGImageAlphaFirst | kCGBitmapByteOrder32Little
* @field version The struct is versioned for future expansion. Pass 0 here.
* @field decode Prior to transformations caused by the colorspace, color channels are subject to a linear transformation.
* This allows for a different range than the typical [0,1.0]. NULL indicates default behavior of [0,1.0]
* range, and is what you should use if you don't understand this parameter. See description of CGImageCreate()
* for a discussion of decode arrays. See also Decode Arrays section of Chapter 4.8 of the PDF specification.
* The vImage_CGImageFormat is not capable of managing the memory held by the decode array. If you created a
* decode array on the heap, you must be sure to release it before all references to it disappear from scope.
*
* @field renderingIntent See CGColorSpace.h. kCGRenderingIntentDefault is typical here. By convention, rendering intent changes that
* are not accompanied by a colorspace change are ignored.
*/
我不明白如何为 YCbCr 像素格式创建 vImage_CGImageFormat。
首先我认为它根本不受支持，但这种格式有特殊功能：Understanding YpCbCr Image Formats and vImageConverter has special functions for multi plane images such as vImageConverter_GetNumberOfSourceBuffers and vImageConverter_GetSourceBufferOrder. In the last function there are many vImage Buffer Type Codes甚至kvImageBufferTypeCode_CVPixelBuffer_YCbCr。
所以，看起来可以为 YCbCr 创建 vImageConverter，我需要帮助来了解如何做。
vImageConverter_CreateWithCGImageFormat 创建用于在两种 Core Graphics 格式之间进行转换的转换器。
在这种情况下，您不应该使用 vImageConverter_CreateForCVToCGImageFormat 将 CVPixelBuffer（带有可以使用 vImageCVImageFormat_CreateWithCVPixelBuffer 生成的 vImageCVImageFormat）转换为CGImage？ vImage_CGImageFormat 是您的目标格式 - 因此其属性由您定义。
有函数 vImageCreateRGBColorSpaceWithPrimariesAndTransferFunction 可以为 YCbCr 像素格式创建适当的颜色 space，但仅用于从 RGB 格式转换或转换为 RGB 格式：
const vImageTransferFunction f709 =
{
.c0 = 1.099,
.c1 = 1.0,
.c2 = 0.0,
.c3 = -0.099,
.gamma = 0.45,
.cutoff = 0.018,
.c4 = 4.5,
.c5 = 0
};
const vImageRGBPrimaries p709 =
{
.red_x = .64, // 1.3 red
.green_x = .30, // 1.3 green
.blue_x = .15, // 1.3 blue
.white_x = 0.3127, // 1.4 white
.red_y = .33, // 1.3 red
.green_y = .60, // 1.3 green
.blue_y = .06, // 1.3 blue
.white_y = 0.3290 // 1.4 white
};
vImage_Error err = kvImageNoError;
CGColorSpaceRef colorSpace = vImageCreateRGBColorSpaceWithPrimariesAndTransferFunction( &p709, &f709, kvImageNoFlags, &err );
vImage_CGImageFormat format = {
.bitsPerComponent = 8,
.bitsPerPixel = 32,
.bitmapInfo = (CGBitmapInfo)kCGImageAlphaNoneSkipFirst | kCGBitmapByteOrderDefault,
.colorSpace = colorSpace
};