AVFoundation 在 iPhone 12 上录制 10 位 HDR 视频
AVFoundation record 10 bit HDR video on iPhone 12
iPhone 12/12 pro 支持以 10 位格式而不是 8 位格式录制杜比视界 HDR 视频，但从 iOS 14.1 SDK 中不清楚 AVCaptureVideoDataOutput 是否支持传送 10 位样本缓冲区可以使用 AVAssetWriter 附加到视频文件。有没有人想过在SDK中是否可行？
编辑：许多应用程序（例如 Apple 的 Clips 应用程序）已开始支持杜比视界 10 位视频录制。但我尝试了所有可用的 API，包括 videoHDREnabled，但它不起作用。所以明确的问题是如何使用 AVFoundation APIs?
录制 HDR（杜比视界）视频
EDIT2：我能够找出支持 10 位像素缓冲区格式的设备格式（'x420'，而不是 420v 或 420f 作为媒体子类型的格式）。在 iPhone 12 mini 上，4 种设备格式支持 kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange 中的 10 位像素缓冲区传送，即使 AVFoundation 文档说这不是受支持的像素格式（引用 - “在 iOS 上，唯一支持的键是 kCVPixelBufferPixelFormatTypeKey。支持的像素格式是 kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange、kCVPixelFormatType_420YpCbCr8BiPlanarFullRange 和 kCVPixelFormatType_32BGRA。”）。下一步是确定是否可以手动选择用于录制的 HDR 格式为 Dolby Vision、HLG 或 HDR10。
如果没有设备在手很难说，但我假设 iPhone 12 中的（部分）AVCaptureDevice 将支持支持 HDR 的 format交货 (isVideoHDRSupported).
相应的 AVCaptureVideoDataOutput 的 availableVideoPixelFormatTypes 可能会列出 kCVPixelFormatType_420YpCbCr10BiPlanarFullRange 和类似的类型作为选项。
11 月 26 日更新。
正如@Deepak 在他自己的回答和评论中发布的那样，'x420' 标记格式将使相机在 HLG 模式下工作。 IP12 Pro 中所有可用的 HLG 启用格式更新如下。
原回答
对于 iOS 14.2，我可以从 AVCaptureDevice 实例中转储所有可用格式，似乎日志输出本身就很清楚。如下所述，希望将 AVCaptureDevice.activeFormat 设置为 HDR+ 宽色格式之一即可。
<AVCaptureDeviceFormat: 0x282d8daf0 'vide'/'x420' 1280x 720, { 1- 30 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:120.00 (upscales @2.91), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports depth>
<AVCaptureDeviceFormat: 0x282d8dac0 'vide'/'x420' 1280x 720, { 1- 60 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:120.00 (upscales @2.91), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports depth>
<AVCaptureDeviceFormat: 0x282d8da50 'vide'/'x420' 1920x1080, { 1- 30 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:120.00 (upscales @1.94), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports depth, supports multicam>
<AVCaptureDeviceFormat: 0x282d8da30 'vide'/'x420' 1920x1080, { 1- 60 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:120.00 (upscales @1.94), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports multicam>
<AVCaptureDeviceFormat: 0x282d8d9e0 'vide'/'x420' 1920x1440, { 1- 30 fps}, HRSI:4032x3024, fov:67.096, max zoom:189.00 (upscales @2.10), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports depth, supports multicam>
<AVCaptureDeviceFormat: 0x282d8d950 'vide'/'x420' 3840x2160, { 1- 30 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:125.25 (upscales @1.00), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports multicam>
截至11月23日，还在调查中，我觉得需要大家共同努力，或者苹果工程师可以看看这个。
我相信我已经观看了关于此主题的所有可用 WWDC17/18/19/20 会议，并且随着新的 iPhone 12 版本的发布，这里有一些发现。
从相机捕获 HDR 并直接保存为 10 位 HLG 视频仅适用于 iPhone 12 及更新版本。这是它在产品发布中声称的内容，我从我朋友的新 phone 那里得到了示例视频，它按预期工作。
在 WWDC2020 中，Export HDR media in your app with AVFoundation 声称：
At this point, I’d like to briefly touch on which Apple platforms can support HDR export.
iOS supports HEVC hardware encoding on devices with Apple A10 Fusion chips or newer.
Fortunately A10 devices have been around for a while, dating back to the iPhone 7, iPads released in 2018, and the 2019 iPod touch.
In regards to Macs, both HEVC and Apple ProRes software encoders are available on all Macs.
HEVC hardware encoding is generally available on 2017 and newer Macs running the new macOS.
Hardware encoding will make the export significantly faster.
同样在该视频中，它声称HDR export only works 10bit HEVC encoding，因此A10+ SoC 应该具有10bit HEVC 编码能力。这是一个猜测，我可以在iPhone 11和SE2上在官方照片应用程序中编辑iPhone12 HLG视频，并且写作性能（4k@60p，HLG）非常好，这是一个很好的选择指标。然而，我没有运气在代码中完成这项工作，视频中列出的示例代码不可能是完整的图片，而且我很难找到一个工作演示。从理论上讲，较旧的设备也应该具有记录 10 位 HLG 的能力，或者相机，thermal/power预算是这里的限制。
However, the only relevant HDR key in this is the VideoProfileLevelKey which must be set to HEVC_Main10_AutoLevel when exporting HDR using the HEVC codec.
Note that 8-bit HEVC HDR is not supported, and this key is not applicable to ProRes exports.
All right, let’s take the time now to summarize how you would configure the keys I just discussed when outputting to two common HDR formats: HLG and HDR10. This table shows what the relevant HDR settings are for exporting an HLG file.
另一个值得反复观看的视频：Edit and play back HDR video with AVFoundation
在测试期间，我确实得到了一个 CVPixelBuffer(format:kCVPixelFormatType_420YpCbCr10BiPlanarFullRange)，它启用了 HDR，并且从示例 HLG 视频中正确地进行了颜色管理。这是我的控制台日志的转储，它适用于任何启用 iOS 14 的设备，即使是相当老的 iPhone6s(A9)，因为它只涉及 10 位 HEVC 解码。
_displayLinkDidRefresh():121 - Optional(<CVPixelBuffer 0x281300500 width=3840 height=2160 pixelFormat=xf20 iosurface=0x282008050 planes=2 poolName=450:decode_1>
<Plane 0 width=3840 height=2160 bytesPerRow=7680>
<Plane 1 width=1920 height=1080 bytesPerRow=7680>
<attributes={
PixelFormatDescription = {
BitsPerComponent = 10;
CGBitmapContextCompatibility = 0;
CGImageCompatibility = 0;
ComponentRange = FullRange;
ContainsAlpha = 0;
ContainsGrayscale = 0;
ContainsRGB = 0;
ContainsYCbCr = 1;
FillExtendedPixelsCallback = {length = 24, bytes = 0x0000000000000000b48ab8a1010000000000000000000000};
IOSurfaceCoreAnimationCompatibility = 1;
IOSurfaceCoreAnimationCompatibilityHTPCOK = 1;
IOSurfaceOpenGLESTextureCompatibility = 1;
OpenGLESCompatibility = 1;
PixelFormat = 2019963440;
Planes = (
{
BitsPerBlock = 16;
HorizontalSubsampling = 1;
VerticalSubsampling = 1;
},
{
BitsPerBlock = 32;
BlackBlock = {length = 4, bytes = 0x00800080};
HorizontalSubsampling = 2;
VerticalSubsampling = 2;
}
);
};
} propagatedAttachments={
CVFieldCount = 1;
CVImageBufferChromaLocationBottomField = Left;
CVImageBufferChromaLocationTopField = Left;
CVImageBufferColorPrimaries = "ITU_R_2020";
CVImageBufferTransferFunction = "ITU_R_2100_HLG";
CVImageBufferYCbCrMatrix = "ITU_R_2020";
QTMovieTime = {
TimeScale = 600;
TimeValue = 12090;
};
} nonPropagatedAttachments={
}>)
好的 none 给出的答案是正确的，所以我在手里抓起 iPhone 12 mini 后进行了研究，这就是我的发现。
AVFoundation 文档是沉默的，有时甚至是不正确的。可以从文档中推断出不可能获得 10 位 HDR 样本缓冲区，特别是如果阅读 videoSettings 属性 of AVCaptureVideoDataOutput[ 的文档=22=]:
On iOS, the only supported key is kCVPixelBufferPixelFormatTypeKey.
Supported pixel formats are kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange,
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange and kCVPixelFormatType_32BGRA
从文档中可以看出，永远无法获得 10 位帧。但是在探测 -[AVCaptureDevice formats] 时，可以找到 4 种不同的格式，并且 mediaSubtype 为 'x420'，即 kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange，一种 10 位格式。在 -[AVCaptureDevice activeFormat] 设置为这 4 种格式之一的时刻，AVCaptureVideoDataOutput 将样本缓冲区格式更改为 kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange！ AVCaptureDevice 的 Active Color Space 也更改为 AVCaptureColorSpace_HLG_BT2020.
如果有人正在寻找这方面的更多信息，我在 iOS AVFoundation 中找到了以下列表，请注意它声明它不支持 CoreVideo 中的所有这些。
/*
CoreVideo pixel format type constants.
CoreVideo does not provide support for all of these formats; this list just defines their names.
*/
public var kCVPixelFormatType_1Monochrome: OSType { get } /* 1 bit indexed */
public var kCVPixelFormatType_2Indexed: OSType { get } /* 2 bit indexed */
public var kCVPixelFormatType_4Indexed: OSType { get } /* 4 bit indexed */
public var kCVPixelFormatType_8Indexed: OSType { get } /* 8 bit indexed */
public var kCVPixelFormatType_1IndexedGray_WhiteIsZero: OSType { get } /* 1 bit indexed gray, white is zero */
public var kCVPixelFormatType_2IndexedGray_WhiteIsZero: OSType { get } /* 2 bit indexed gray, white is zero */
public var kCVPixelFormatType_4IndexedGray_WhiteIsZero: OSType { get } /* 4 bit indexed gray, white is zero */
public var kCVPixelFormatType_8IndexedGray_WhiteIsZero: OSType { get } /* 8 bit indexed gray, white is zero */
public var kCVPixelFormatType_16BE555: OSType { get } /* 16 bit BE RGB 555 */
public var kCVPixelFormatType_16LE555: OSType { get } /* 16 bit LE RGB 555 */
public var kCVPixelFormatType_16LE5551: OSType { get } /* 16 bit LE RGB 5551 */
public var kCVPixelFormatType_16BE565: OSType { get } /* 16 bit BE RGB 565 */
public var kCVPixelFormatType_16LE565: OSType { get } /* 16 bit LE RGB 565 */
public var kCVPixelFormatType_24RGB: OSType { get } /* 24 bit RGB */
public var kCVPixelFormatType_24BGR: OSType { get } /* 24 bit BGR */
public var kCVPixelFormatType_32ARGB: OSType { get } /* 32 bit ARGB */
public var kCVPixelFormatType_32BGRA: OSType { get } /* 32 bit BGRA */
public var kCVPixelFormatType_32ABGR: OSType { get } /* 32 bit ABGR */
public var kCVPixelFormatType_32RGBA: OSType { get } /* 32 bit RGBA */
public var kCVPixelFormatType_64ARGB: OSType { get } /* 64 bit ARGB, 16-bit big-endian samples */
public var kCVPixelFormatType_64RGBALE: OSType { get } /* 64 bit RGBA, 16-bit little-endian full-range (0-65535) samples */
public var kCVPixelFormatType_48RGB: OSType { get } /* 48 bit RGB, 16-bit big-endian samples */
public var kCVPixelFormatType_32AlphaGray: OSType { get } /* 32 bit AlphaGray, 16-bit big-endian samples, black is zero */
public var kCVPixelFormatType_16Gray: OSType { get } /* 16 bit Grayscale, 16-bit big-endian samples, black is zero */
public var kCVPixelFormatType_30RGB: OSType { get } /* 30 bit RGB, 10-bit big-endian samples, 2 unused padding bits (at least significant end). */
public var kCVPixelFormatType_422YpCbCr8: OSType { get } /* Component Y'CbCr 8-bit 4:2:2, ordered Cb Y'0 Cr Y'1 */
public var kCVPixelFormatType_4444YpCbCrA8: OSType { get } /* Component Y'CbCrA 8-bit 4:4:4:4, ordered Cb Y' Cr A */
public var kCVPixelFormatType_4444YpCbCrA8R: OSType { get } /* Component Y'CbCrA 8-bit 4:4:4:4, rendering format. full range alpha, zero biased YUV, ordered A Y' Cb Cr */
public var kCVPixelFormatType_4444AYpCbCr8: OSType { get } /* Component Y'CbCrA 8-bit 4:4:4:4, ordered A Y' Cb Cr, full range alpha, video range Y'CbCr. */
public var kCVPixelFormatType_4444AYpCbCr16: OSType { get } /* Component Y'CbCrA 16-bit 4:4:4:4, ordered A Y' Cb Cr, full range alpha, video range Y'CbCr, 16-bit little-endian samples. */
public var kCVPixelFormatType_444YpCbCr8: OSType { get } /* Component Y'CbCr 8-bit 4:4:4 */
public var kCVPixelFormatType_422YpCbCr16: OSType { get } /* Component Y'CbCr 10,12,14,16-bit 4:2:2 */
public var kCVPixelFormatType_422YpCbCr10: OSType { get } /* Component Y'CbCr 10-bit 4:2:2 */
public var kCVPixelFormatType_444YpCbCr10: OSType { get } /* Component Y'CbCr 10-bit 4:4:4 */
public var kCVPixelFormatType_420YpCbCr8Planar: OSType { get } /* Planar Component Y'CbCr 8-bit 4:2:0. baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrPlanar struct */
public var kCVPixelFormatType_420YpCbCr8PlanarFullRange: OSType { get } /* Planar Component Y'CbCr 8-bit 4:2:0, full range. baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrPlanar struct */
public var kCVPixelFormatType_422YpCbCr_4A_8BiPlanar: OSType { get } /* First plane: Video-range Component Y'CbCr 8-bit 4:2:2, ordered Cb Y'0 Cr Y'1; second plane: alpha 8-bit 0-255 */
public var kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange: OSType { get } /* Bi-Planar Component Y'CbCr 8-bit 4:2:0, video-range (luma=[16,235] chroma=[16,240]). baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
public var kCVPixelFormatType_420YpCbCr8BiPlanarFullRange: OSType { get } /* Bi-Planar Component Y'CbCr 8-bit 4:2:0, full-range (luma=[0,255] chroma=[1,255]). baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
public var kCVPixelFormatType_422YpCbCr8BiPlanarVideoRange: OSType { get } /* Bi-Planar Component Y'CbCr 8-bit 4:2:2, video-range (luma=[16,235] chroma=[16,240]). baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
public var kCVPixelFormatType_422YpCbCr8BiPlanarFullRange: OSType { get } /* Bi-Planar Component Y'CbCr 8-bit 4:2:2, full-range (luma=[0,255] chroma=[1,255]). baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
public var kCVPixelFormatType_444YpCbCr8BiPlanarVideoRange: OSType { get } /* Bi-Planar Component Y'CbCr 8-bit 4:4:4, video-range (luma=[16,235] chroma=[16,240]). baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
public var kCVPixelFormatType_444YpCbCr8BiPlanarFullRange: OSType { get } /* Bi-Planar Component Y'CbCr 8-bit 4:4:4, full-range (luma=[0,255] chroma=[1,255]). baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
public var kCVPixelFormatType_422YpCbCr8_yuvs: OSType { get } /* Component Y'CbCr 8-bit 4:2:2, ordered Y'0 Cb Y'1 Cr */
public var kCVPixelFormatType_422YpCbCr8FullRange: OSType { get } /* Component Y'CbCr 8-bit 4:2:2, full range, ordered Y'0 Cb Y'1 Cr */
public var kCVPixelFormatType_OneComponent8: OSType { get } /* 8 bit one component, black is zero */
public var kCVPixelFormatType_TwoComponent8: OSType { get } /* 8 bit two component, black is zero */
public var kCVPixelFormatType_30RGBLEPackedWideGamut: OSType { get } /* little-endian RGB101010, 2 MSB are zero, wide-gamut (384-895) */
public var kCVPixelFormatType_ARGB2101010LEPacked: OSType { get } /* little-endian ARGB2101010 full-range ARGB */
public var kCVPixelFormatType_OneComponent10: OSType { get } /* 10 bit little-endian one component, stored as 10 MSBs of 16 bits, black is zero */
public var kCVPixelFormatType_OneComponent12: OSType { get } /* 12 bit little-endian one component, stored as 12 MSBs of 16 bits, black is zero */
public var kCVPixelFormatType_OneComponent16: OSType { get } /* 16 bit little-endian one component, black is zero */
public var kCVPixelFormatType_TwoComponent16: OSType { get } /* 16 bit little-endian two component, black is zero */
public var kCVPixelFormatType_OneComponent16Half: OSType { get } /* 16 bit one component IEEE half-precision float, 16-bit little-endian samples */
public var kCVPixelFormatType_OneComponent32Float: OSType { get } /* 32 bit one component IEEE float, 32-bit little-endian samples */
public var kCVPixelFormatType_TwoComponent16Half: OSType { get } /* 16 bit two component IEEE half-precision float, 16-bit little-endian samples */
public var kCVPixelFormatType_TwoComponent32Float: OSType { get } /* 32 bit two component IEEE float, 32-bit little-endian samples */
public var kCVPixelFormatType_64RGBAHalf: OSType { get } /* 64 bit RGBA IEEE half-precision float, 16-bit little-endian samples */
public var kCVPixelFormatType_128RGBAFloat: OSType { get } /* 128 bit RGBA IEEE float, 32-bit little-endian samples */
public var kCVPixelFormatType_14Bayer_GRBG: OSType { get } /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered G R G R... alternating with B G B G... */
public var kCVPixelFormatType_14Bayer_RGGB: OSType { get } /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered R G R G... alternating with G B G B... */
public var kCVPixelFormatType_14Bayer_BGGR: OSType { get } /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered B G B G... alternating with G R G R... */
public var kCVPixelFormatType_14Bayer_GBRG: OSType { get } /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered G B G B... alternating with R G R G... */
public var kCVPixelFormatType_DisparityFloat16: OSType { get } /* IEEE754-2008 binary16 (half float), describing the normalized shift when comparing two images. Units are 1/meters: ( pixelShift / (pixelFocalLength * baselineInMeters) ) */
public var kCVPixelFormatType_DisparityFloat32: OSType { get } /* IEEE754-2008 binary32 float, describing the normalized shift when comparing two images. Units are 1/meters: ( pixelShift / (pixelFocalLength * baselineInMeters) ) */
public var kCVPixelFormatType_DepthFloat16: OSType { get } /* IEEE754-2008 binary16 (half float), describing the depth (distance to an object) in meters */
public var kCVPixelFormatType_DepthFloat32: OSType { get } /* IEEE754-2008 binary32 float, describing the depth (distance to an object) in meters */
public var kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange: OSType { get } /* 2 plane YCbCr10 4:2:0, each 10 bits in the MSBs of 16bits, video-range (luma=[64,940] chroma=[64,960]) */
public var kCVPixelFormatType_422YpCbCr10BiPlanarVideoRange: OSType { get } /* 2 plane YCbCr10 4:2:2, each 10 bits in the MSBs of 16bits, video-range (luma=[64,940] chroma=[64,960]) */
public var kCVPixelFormatType_444YpCbCr10BiPlanarVideoRange: OSType { get } /* 2 plane YCbCr10 4:4:4, each 10 bits in the MSBs of 16bits, video-range (luma=[64,940] chroma=[64,960]) */
public var kCVPixelFormatType_420YpCbCr10BiPlanarFullRange: OSType { get } /* 2 plane YCbCr10 4:2:0, each 10 bits in the MSBs of 16bits, full-range (Y range 0-1023) */
public var kCVPixelFormatType_422YpCbCr10BiPlanarFullRange: OSType { get } /* 2 plane YCbCr10 4:2:2, each 10 bits in the MSBs of 16bits, full-range (Y range 0-1023) */
public var kCVPixelFormatType_444YpCbCr10BiPlanarFullRange: OSType { get } /* 2 plane YCbCr10 4:4:4, each 10 bits in the MSBs of 16bits, full-range (Y range 0-1023) */
public var kCVPixelFormatType_420YpCbCr8VideoRange_8A_TriPlanar: OSType { get } /* first and second planes as per 420YpCbCr8BiPlanarVideoRange (420v), alpha 8 bits in third plane full-range. No CVPlanarPixelBufferInfo struct. */
public var kCVPixelFormatType_16VersatileBayer: OSType { get } /* Single plane Bayer 16-bit little-endian sensor element ("sensel") samples from full-size decoding of ProRes RAW images; Bayer pattern (sensel ordering) and other raw conversion information is described via buffer attachments */
public var kCVPixelFormatType_64RGBA_DownscaledProResRAW: OSType { get } /* Single plane 64-bit RGBA (16-bit little-endian samples) from downscaled decoding of ProRes RAW images; components--which may not be co-sited with one another--are sensel values and require raw conversion, information for which is described via buffer attachments */
强制使用 BT2020 格式是确保您使用杜比视界拍摄的正确方法。 iOS 需要 14.1 或更高版本才能执行此操作。这是我如何执行此操作的简短片段：
// Setup your session
session.beginConfiguration()
session.sessionPreset = .hd1280x720
// Add your camera to the session
let camera = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .front)
let cameraInput = try AVCaptureDeviceInput(device: camera)
session.addInput(cameraInput)
// Important! Commit the session configuration before configuring your camera
session.commitConfiguration()
// Configure camera
try camera.lockForConfiguration()
// Force HDR on
camera.automaticallyAdjustsVideoHDREnabled = false
camera.isVideoHDREnabled = true
// Find the first 720p format that supports the correct colorspace
let desiredColorSpace = AVCaptureColorSpace.HLG_BT2020
let desiredFormat = camera.formats.first { format in
// You could of course choose a different resolution if desired
format.formatDescription.dimensions == CMVideoDimensions(width: 1280, height: 720) &&
format.supportedColorSpaces.contains(desiredColorSpace)
}
// Set the HDR format
if let format = desiredFormat {
camera.activeFormat = format
camera.activeColorSpace = desiredColorSpace
} else {
assertionFailure("Counldn't find HDR camera format")
}
camera.unlockForConfiguration()
iPhone 12/12 pro 支持以 10 位格式而不是 8 位格式录制杜比视界 HDR 视频，但从 iOS 14.1 SDK 中不清楚 AVCaptureVideoDataOutput 是否支持传送 10 位样本缓冲区可以使用 AVAssetWriter 附加到视频文件。有没有人想过在SDK中是否可行？
编辑：许多应用程序（例如 Apple 的 Clips 应用程序）已开始支持杜比视界 10 位视频录制。但我尝试了所有可用的 API，包括 videoHDREnabled，但它不起作用。所以明确的问题是如何使用 AVFoundation APIs?
录制 HDR（杜比视界）视频EDIT2：我能够找出支持 10 位像素缓冲区格式的设备格式（'x420'，而不是 420v 或 420f 作为媒体子类型的格式）。在 iPhone 12 mini 上，4 种设备格式支持 kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange 中的 10 位像素缓冲区传送，即使 AVFoundation 文档说这不是受支持的像素格式（引用 - “在 iOS 上，唯一支持的键是 kCVPixelBufferPixelFormatTypeKey。支持的像素格式是 kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange、kCVPixelFormatType_420YpCbCr8BiPlanarFullRange 和 kCVPixelFormatType_32BGRA。”）。下一步是确定是否可以手动选择用于录制的 HDR 格式为 Dolby Vision、HLG 或 HDR10。
如果没有设备在手很难说，但我假设 iPhone 12 中的（部分）AVCaptureDevice 将支持支持 HDR 的 format交货 (isVideoHDRSupported).
相应的 AVCaptureVideoDataOutput 的 availableVideoPixelFormatTypes 可能会列出 kCVPixelFormatType_420YpCbCr10BiPlanarFullRange 和类似的类型作为选项。
11 月 26 日更新。
正如@Deepak 在他自己的回答和评论中发布的那样，'x420' 标记格式将使相机在 HLG 模式下工作。 IP12 Pro 中所有可用的 HLG 启用格式更新如下。
原回答
对于 iOS 14.2，我可以从 AVCaptureDevice 实例中转储所有可用格式，似乎日志输出本身就很清楚。如下所述，希望将 AVCaptureDevice.activeFormat 设置为 HDR+ 宽色格式之一即可。
<AVCaptureDeviceFormat: 0x282d8daf0 'vide'/'x420' 1280x 720, { 1- 30 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:120.00 (upscales @2.91), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports depth>
<AVCaptureDeviceFormat: 0x282d8dac0 'vide'/'x420' 1280x 720, { 1- 60 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:120.00 (upscales @2.91), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports depth>
<AVCaptureDeviceFormat: 0x282d8da50 'vide'/'x420' 1920x1080, { 1- 30 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:120.00 (upscales @1.94), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports depth, supports multicam>
<AVCaptureDeviceFormat: 0x282d8da30 'vide'/'x420' 1920x1080, { 1- 60 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:120.00 (upscales @1.94), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports multicam>
<AVCaptureDeviceFormat: 0x282d8d9e0 'vide'/'x420' 1920x1440, { 1- 30 fps}, HRSI:4032x3024, fov:67.096, max zoom:189.00 (upscales @2.10), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports depth, supports multicam>
<AVCaptureDeviceFormat: 0x282d8d950 'vide'/'x420' 3840x2160, { 1- 30 fps}, HRSI:4096x2304, fov:68.161, supports vis, max zoom:125.25 (upscales @1.00), AF System:2, ISO:34.0-3264.0, SS:0.000014-1.000000, supports wide color, supports multicam>
截至11月23日，还在调查中，我觉得需要大家共同努力，或者苹果工程师可以看看这个。
我相信我已经观看了关于此主题的所有可用 WWDC17/18/19/20 会议，并且随着新的 iPhone 12 版本的发布，这里有一些发现。
从相机捕获 HDR 并直接保存为 10 位 HLG 视频仅适用于 iPhone 12 及更新版本。这是它在产品发布中声称的内容，我从我朋友的新 phone 那里得到了示例视频，它按预期工作。
在 WWDC2020 中，Export HDR media in your app with AVFoundation 声称：
At this point, I’d like to briefly touch on which Apple platforms can support HDR export.
iOS supports HEVC hardware encoding on devices with Apple A10 Fusion chips or newer.
Fortunately A10 devices have been around for a while, dating back to the iPhone 7, iPads released in 2018, and the 2019 iPod touch.
In regards to Macs, both HEVC and Apple ProRes software encoders are available on all Macs.
HEVC hardware encoding is generally available on 2017 and newer Macs running the new macOS.
Hardware encoding will make the export significantly faster.
同样在该视频中，它声称HDR export only works 10bit HEVC encoding，因此A10+ SoC 应该具有10bit HEVC 编码能力。这是一个猜测，我可以在iPhone 11和SE2上在官方照片应用程序中编辑iPhone12 HLG视频，并且写作性能（4k@60p，HLG）非常好，这是一个很好的选择指标。然而，我没有运气在代码中完成这项工作，视频中列出的示例代码不可能是完整的图片，而且我很难找到一个工作演示。从理论上讲，较旧的设备也应该具有记录 10 位 HLG 的能力，或者相机，thermal/power预算是这里的限制。
However, the only relevant HDR key in this is the VideoProfileLevelKey which must be set to HEVC_Main10_AutoLevel when exporting HDR using the HEVC codec.
Note that 8-bit HEVC HDR is not supported, and this key is not applicable to ProRes exports.
All right, let’s take the time now to summarize how you would configure the keys I just discussed when outputting to two common HDR formats: HLG and HDR10. This table shows what the relevant HDR settings are for exporting an HLG file.
另一个值得反复观看的视频：Edit and play back HDR video with AVFoundation
在测试期间，我确实得到了一个 CVPixelBuffer(format:kCVPixelFormatType_420YpCbCr10BiPlanarFullRange)，它启用了 HDR，并且从示例 HLG 视频中正确地进行了颜色管理。这是我的控制台日志的转储，它适用于任何启用 iOS 14 的设备，即使是相当老的 iPhone6s(A9)，因为它只涉及 10 位 HEVC 解码。
_displayLinkDidRefresh():121 - Optional(<CVPixelBuffer 0x281300500 width=3840 height=2160 pixelFormat=xf20 iosurface=0x282008050 planes=2 poolName=450:decode_1>
<Plane 0 width=3840 height=2160 bytesPerRow=7680>
<Plane 1 width=1920 height=1080 bytesPerRow=7680>
<attributes={
PixelFormatDescription = {
BitsPerComponent = 10;
CGBitmapContextCompatibility = 0;
CGImageCompatibility = 0;
ComponentRange = FullRange;
ContainsAlpha = 0;
ContainsGrayscale = 0;
ContainsRGB = 0;
ContainsYCbCr = 1;
FillExtendedPixelsCallback = {length = 24, bytes = 0x0000000000000000b48ab8a1010000000000000000000000};
IOSurfaceCoreAnimationCompatibility = 1;
IOSurfaceCoreAnimationCompatibilityHTPCOK = 1;
IOSurfaceOpenGLESTextureCompatibility = 1;
OpenGLESCompatibility = 1;
PixelFormat = 2019963440;
Planes = (
{
BitsPerBlock = 16;
HorizontalSubsampling = 1;
VerticalSubsampling = 1;
},
{
BitsPerBlock = 32;
BlackBlock = {length = 4, bytes = 0x00800080};
HorizontalSubsampling = 2;
VerticalSubsampling = 2;
}
);
};
} propagatedAttachments={
CVFieldCount = 1;
CVImageBufferChromaLocationBottomField = Left;
CVImageBufferChromaLocationTopField = Left;
CVImageBufferColorPrimaries = "ITU_R_2020";
CVImageBufferTransferFunction = "ITU_R_2100_HLG";
CVImageBufferYCbCrMatrix = "ITU_R_2020";
QTMovieTime = {
TimeScale = 600;
TimeValue = 12090;
};
} nonPropagatedAttachments={
}>)
好的 none 给出的答案是正确的，所以我在手里抓起 iPhone 12 mini 后进行了研究，这就是我的发现。
AVFoundation 文档是沉默的，有时甚至是不正确的。可以从文档中推断出不可能获得 10 位 HDR 样本缓冲区，特别是如果阅读 videoSettings 属性 of AVCaptureVideoDataOutput[ 的文档=22=]:
On iOS, the only supported key is kCVPixelBufferPixelFormatTypeKey.
Supported pixel formats are kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange,
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange and kCVPixelFormatType_32BGRA
从文档中可以看出，永远无法获得 10 位帧。但是在探测 -[AVCaptureDevice formats] 时，可以找到 4 种不同的格式，并且 mediaSubtype 为 'x420'，即 kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange，一种 10 位格式。在 -[AVCaptureDevice activeFormat] 设置为这 4 种格式之一的时刻，AVCaptureVideoDataOutput 将样本缓冲区格式更改为 kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange！ AVCaptureDevice 的 Active Color Space 也更改为 AVCaptureColorSpace_HLG_BT2020.
如果有人正在寻找这方面的更多信息，我在 iOS AVFoundation 中找到了以下列表，请注意它声明它不支持 CoreVideo 中的所有这些。
/*
CoreVideo pixel format type constants.
CoreVideo does not provide support for all of these formats; this list just defines their names.
*/
public var kCVPixelFormatType_1Monochrome: OSType { get } /* 1 bit indexed */
public var kCVPixelFormatType_2Indexed: OSType { get } /* 2 bit indexed */
public var kCVPixelFormatType_4Indexed: OSType { get } /* 4 bit indexed */
public var kCVPixelFormatType_8Indexed: OSType { get } /* 8 bit indexed */
public var kCVPixelFormatType_1IndexedGray_WhiteIsZero: OSType { get } /* 1 bit indexed gray, white is zero */
public var kCVPixelFormatType_2IndexedGray_WhiteIsZero: OSType { get } /* 2 bit indexed gray, white is zero */
public var kCVPixelFormatType_4IndexedGray_WhiteIsZero: OSType { get } /* 4 bit indexed gray, white is zero */
public var kCVPixelFormatType_8IndexedGray_WhiteIsZero: OSType { get } /* 8 bit indexed gray, white is zero */
public var kCVPixelFormatType_16BE555: OSType { get } /* 16 bit BE RGB 555 */
public var kCVPixelFormatType_16LE555: OSType { get } /* 16 bit LE RGB 555 */
public var kCVPixelFormatType_16LE5551: OSType { get } /* 16 bit LE RGB 5551 */
public var kCVPixelFormatType_16BE565: OSType { get } /* 16 bit BE RGB 565 */
public var kCVPixelFormatType_16LE565: OSType { get } /* 16 bit LE RGB 565 */
public var kCVPixelFormatType_24RGB: OSType { get } /* 24 bit RGB */
public var kCVPixelFormatType_24BGR: OSType { get } /* 24 bit BGR */
public var kCVPixelFormatType_32ARGB: OSType { get } /* 32 bit ARGB */
public var kCVPixelFormatType_32BGRA: OSType { get } /* 32 bit BGRA */
public var kCVPixelFormatType_32ABGR: OSType { get } /* 32 bit ABGR */
public var kCVPixelFormatType_32RGBA: OSType { get } /* 32 bit RGBA */
public var kCVPixelFormatType_64ARGB: OSType { get } /* 64 bit ARGB, 16-bit big-endian samples */
public var kCVPixelFormatType_64RGBALE: OSType { get } /* 64 bit RGBA, 16-bit little-endian full-range (0-65535) samples */
public var kCVPixelFormatType_48RGB: OSType { get } /* 48 bit RGB, 16-bit big-endian samples */
public var kCVPixelFormatType_32AlphaGray: OSType { get } /* 32 bit AlphaGray, 16-bit big-endian samples, black is zero */
public var kCVPixelFormatType_16Gray: OSType { get } /* 16 bit Grayscale, 16-bit big-endian samples, black is zero */
public var kCVPixelFormatType_30RGB: OSType { get } /* 30 bit RGB, 10-bit big-endian samples, 2 unused padding bits (at least significant end). */
public var kCVPixelFormatType_422YpCbCr8: OSType { get } /* Component Y'CbCr 8-bit 4:2:2, ordered Cb Y'0 Cr Y'1 */
public var kCVPixelFormatType_4444YpCbCrA8: OSType { get } /* Component Y'CbCrA 8-bit 4:4:4:4, ordered Cb Y' Cr A */
public var kCVPixelFormatType_4444YpCbCrA8R: OSType { get } /* Component Y'CbCrA 8-bit 4:4:4:4, rendering format. full range alpha, zero biased YUV, ordered A Y' Cb Cr */
public var kCVPixelFormatType_4444AYpCbCr8: OSType { get } /* Component Y'CbCrA 8-bit 4:4:4:4, ordered A Y' Cb Cr, full range alpha, video range Y'CbCr. */
public var kCVPixelFormatType_4444AYpCbCr16: OSType { get } /* Component Y'CbCrA 16-bit 4:4:4:4, ordered A Y' Cb Cr, full range alpha, video range Y'CbCr, 16-bit little-endian samples. */
public var kCVPixelFormatType_444YpCbCr8: OSType { get } /* Component Y'CbCr 8-bit 4:4:4 */
public var kCVPixelFormatType_422YpCbCr16: OSType { get } /* Component Y'CbCr 10,12,14,16-bit 4:2:2 */
public var kCVPixelFormatType_422YpCbCr10: OSType { get } /* Component Y'CbCr 10-bit 4:2:2 */
public var kCVPixelFormatType_444YpCbCr10: OSType { get } /* Component Y'CbCr 10-bit 4:4:4 */
public var kCVPixelFormatType_420YpCbCr8Planar: OSType { get } /* Planar Component Y'CbCr 8-bit 4:2:0. baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrPlanar struct */
public var kCVPixelFormatType_420YpCbCr8PlanarFullRange: OSType { get } /* Planar Component Y'CbCr 8-bit 4:2:0, full range. baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrPlanar struct */
public var kCVPixelFormatType_422YpCbCr_4A_8BiPlanar: OSType { get } /* First plane: Video-range Component Y'CbCr 8-bit 4:2:2, ordered Cb Y'0 Cr Y'1; second plane: alpha 8-bit 0-255 */
public var kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange: OSType { get } /* Bi-Planar Component Y'CbCr 8-bit 4:2:0, video-range (luma=[16,235] chroma=[16,240]). baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
public var kCVPixelFormatType_420YpCbCr8BiPlanarFullRange: OSType { get } /* Bi-Planar Component Y'CbCr 8-bit 4:2:0, full-range (luma=[0,255] chroma=[1,255]). baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
public var kCVPixelFormatType_422YpCbCr8BiPlanarVideoRange: OSType { get } /* Bi-Planar Component Y'CbCr 8-bit 4:2:2, video-range (luma=[16,235] chroma=[16,240]). baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
public var kCVPixelFormatType_422YpCbCr8BiPlanarFullRange: OSType { get } /* Bi-Planar Component Y'CbCr 8-bit 4:2:2, full-range (luma=[0,255] chroma=[1,255]). baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
public var kCVPixelFormatType_444YpCbCr8BiPlanarVideoRange: OSType { get } /* Bi-Planar Component Y'CbCr 8-bit 4:4:4, video-range (luma=[16,235] chroma=[16,240]). baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
public var kCVPixelFormatType_444YpCbCr8BiPlanarFullRange: OSType { get } /* Bi-Planar Component Y'CbCr 8-bit 4:4:4, full-range (luma=[0,255] chroma=[1,255]). baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
public var kCVPixelFormatType_422YpCbCr8_yuvs: OSType { get } /* Component Y'CbCr 8-bit 4:2:2, ordered Y'0 Cb Y'1 Cr */
public var kCVPixelFormatType_422YpCbCr8FullRange: OSType { get } /* Component Y'CbCr 8-bit 4:2:2, full range, ordered Y'0 Cb Y'1 Cr */
public var kCVPixelFormatType_OneComponent8: OSType { get } /* 8 bit one component, black is zero */
public var kCVPixelFormatType_TwoComponent8: OSType { get } /* 8 bit two component, black is zero */
public var kCVPixelFormatType_30RGBLEPackedWideGamut: OSType { get } /* little-endian RGB101010, 2 MSB are zero, wide-gamut (384-895) */
public var kCVPixelFormatType_ARGB2101010LEPacked: OSType { get } /* little-endian ARGB2101010 full-range ARGB */
public var kCVPixelFormatType_OneComponent10: OSType { get } /* 10 bit little-endian one component, stored as 10 MSBs of 16 bits, black is zero */
public var kCVPixelFormatType_OneComponent12: OSType { get } /* 12 bit little-endian one component, stored as 12 MSBs of 16 bits, black is zero */
public var kCVPixelFormatType_OneComponent16: OSType { get } /* 16 bit little-endian one component, black is zero */
public var kCVPixelFormatType_TwoComponent16: OSType { get } /* 16 bit little-endian two component, black is zero */
public var kCVPixelFormatType_OneComponent16Half: OSType { get } /* 16 bit one component IEEE half-precision float, 16-bit little-endian samples */
public var kCVPixelFormatType_OneComponent32Float: OSType { get } /* 32 bit one component IEEE float, 32-bit little-endian samples */
public var kCVPixelFormatType_TwoComponent16Half: OSType { get } /* 16 bit two component IEEE half-precision float, 16-bit little-endian samples */
public var kCVPixelFormatType_TwoComponent32Float: OSType { get } /* 32 bit two component IEEE float, 32-bit little-endian samples */
public var kCVPixelFormatType_64RGBAHalf: OSType { get } /* 64 bit RGBA IEEE half-precision float, 16-bit little-endian samples */
public var kCVPixelFormatType_128RGBAFloat: OSType { get } /* 128 bit RGBA IEEE float, 32-bit little-endian samples */
public var kCVPixelFormatType_14Bayer_GRBG: OSType { get } /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered G R G R... alternating with B G B G... */
public var kCVPixelFormatType_14Bayer_RGGB: OSType { get } /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered R G R G... alternating with G B G B... */
public var kCVPixelFormatType_14Bayer_BGGR: OSType { get } /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered B G B G... alternating with G R G R... */
public var kCVPixelFormatType_14Bayer_GBRG: OSType { get } /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered G B G B... alternating with R G R G... */
public var kCVPixelFormatType_DisparityFloat16: OSType { get } /* IEEE754-2008 binary16 (half float), describing the normalized shift when comparing two images. Units are 1/meters: ( pixelShift / (pixelFocalLength * baselineInMeters) ) */
public var kCVPixelFormatType_DisparityFloat32: OSType { get } /* IEEE754-2008 binary32 float, describing the normalized shift when comparing two images. Units are 1/meters: ( pixelShift / (pixelFocalLength * baselineInMeters) ) */
public var kCVPixelFormatType_DepthFloat16: OSType { get } /* IEEE754-2008 binary16 (half float), describing the depth (distance to an object) in meters */
public var kCVPixelFormatType_DepthFloat32: OSType { get } /* IEEE754-2008 binary32 float, describing the depth (distance to an object) in meters */
public var kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange: OSType { get } /* 2 plane YCbCr10 4:2:0, each 10 bits in the MSBs of 16bits, video-range (luma=[64,940] chroma=[64,960]) */
public var kCVPixelFormatType_422YpCbCr10BiPlanarVideoRange: OSType { get } /* 2 plane YCbCr10 4:2:2, each 10 bits in the MSBs of 16bits, video-range (luma=[64,940] chroma=[64,960]) */
public var kCVPixelFormatType_444YpCbCr10BiPlanarVideoRange: OSType { get } /* 2 plane YCbCr10 4:4:4, each 10 bits in the MSBs of 16bits, video-range (luma=[64,940] chroma=[64,960]) */
public var kCVPixelFormatType_420YpCbCr10BiPlanarFullRange: OSType { get } /* 2 plane YCbCr10 4:2:0, each 10 bits in the MSBs of 16bits, full-range (Y range 0-1023) */
public var kCVPixelFormatType_422YpCbCr10BiPlanarFullRange: OSType { get } /* 2 plane YCbCr10 4:2:2, each 10 bits in the MSBs of 16bits, full-range (Y range 0-1023) */
public var kCVPixelFormatType_444YpCbCr10BiPlanarFullRange: OSType { get } /* 2 plane YCbCr10 4:4:4, each 10 bits in the MSBs of 16bits, full-range (Y range 0-1023) */
public var kCVPixelFormatType_420YpCbCr8VideoRange_8A_TriPlanar: OSType { get } /* first and second planes as per 420YpCbCr8BiPlanarVideoRange (420v), alpha 8 bits in third plane full-range. No CVPlanarPixelBufferInfo struct. */
public var kCVPixelFormatType_16VersatileBayer: OSType { get } /* Single plane Bayer 16-bit little-endian sensor element ("sensel") samples from full-size decoding of ProRes RAW images; Bayer pattern (sensel ordering) and other raw conversion information is described via buffer attachments */
public var kCVPixelFormatType_64RGBA_DownscaledProResRAW: OSType { get } /* Single plane 64-bit RGBA (16-bit little-endian samples) from downscaled decoding of ProRes RAW images; components--which may not be co-sited with one another--are sensel values and require raw conversion, information for which is described via buffer attachments */
强制使用 BT2020 格式是确保您使用杜比视界拍摄的正确方法。 iOS 需要 14.1 或更高版本才能执行此操作。这是我如何执行此操作的简短片段：
// Setup your session
session.beginConfiguration()
session.sessionPreset = .hd1280x720
// Add your camera to the session
let camera = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .front)
let cameraInput = try AVCaptureDeviceInput(device: camera)
session.addInput(cameraInput)
// Important! Commit the session configuration before configuring your camera
session.commitConfiguration()
// Configure camera
try camera.lockForConfiguration()
// Force HDR on
camera.automaticallyAdjustsVideoHDREnabled = false
camera.isVideoHDREnabled = true
// Find the first 720p format that supports the correct colorspace
let desiredColorSpace = AVCaptureColorSpace.HLG_BT2020
let desiredFormat = camera.formats.first { format in
// You could of course choose a different resolution if desired
format.formatDescription.dimensions == CMVideoDimensions(width: 1280, height: 720) &&
format.supportedColorSpaces.contains(desiredColorSpace)
}
// Set the HDR format
if let format = desiredFormat {
camera.activeFormat = format
camera.activeColorSpace = desiredColorSpace
} else {
assertionFailure("Counldn't find HDR camera format")
}
camera.unlockForConfiguration()