如何从 VideoMediaFrame.Direct3DSurface 获取像素数据?
How to get pixel data from VideoMediaFrame.Direct3DSurface?
我正在为 Micorsoft HoloLens 2 创建 Unity 应用。
该应用使用 Windows.Media.Capture.MediaCapture 和 Microsoft.MixedReality.WebRTC.[=13= 捕捉和分享相机视频帧]
我在 MediaFrameReader.FrameArrived[= 上获得了由“NV12”格式化的 VideoMediaFrame.Direct3DSurface(Windows.Graphics.DirectX.Direct3D11.IDirect3DSurface) 42=]事件。
但我不知道如何从 IDirect3DSurface 对象获取像素数据。
我在下面试过:
C#代码
[DllImport("Direct3DSurfaceAccess")]
private static extern void Direct3DSurfaceAccess([MarshalAs(UnmanagedType.Interface)]Windows.Graphics.DirectX.Direct3D11.IDirect3DSurface d3dSurface);
// MediaFrameReader.FrameArrived event.
private void Nv12FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args) {
using var mediaFrameReference = sender.TryAcquireLatestFrame();
// I could't find how to get pixel data from IDirect3DSurface using C# only.
// Use C++/WinRT for access IDirect3DSurface as below.
Direct3DSurfaceAccess(mediaFrameReference.VideoMediaFrame.Direct3DSurface);
}
C++代码
void __stdcall Direct3DSurfaceAccess(const winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DSurface* d3dSurface) {
// Here, confirmed that d3dSurface is not null.
try {
// Issue is here.
winrt::com_ptr<::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess> dxgiInterfaceAccess;
d3dSurface->try_as(dxgiInterfaceAccess);
// App crashes on d3dSurface->try_as.
// Call other d3dSurface method as a test.
winrt::Windows::Graphics::DirectX::Direct3D11::Direct3DSurfaceDescription desc = d3dSurface->Description();
// Same above. App crashes on d3dSurface->Description.
// Unreachable to here...
// When succeeded to get the IDirect3DDxgiInterfaceAccess, read pixel data by below code.
winrt::com_ptr<::IDXGISurface> nativeSurface;
dxgiInterfaceAccess->GetInterface(IID_PPV_ARGS(nativeSurface.put()));
::DXGI_MAPPED_RECT rect;
nativeSurface->Map(&rect, DXGI_MAP_READ);
const BYTE* pixels = rect.pBits; // Read pixel data from the pointer.
nativeSurface->Unmap();
} catch (...) {
// Can't catch any exceptions.
// App freezes and downs on HoloLens 2.
}
}
我认为如何编组是错误的 IDirect3DSurface。
如何从 IDirect3DSurface 获取像素数据?
您不能在 dll 边界定义 C++/WinRT 对象指针,因为在二进制级别,它不是 COM 对象(因此 C# 正在传递映射到 C++/WinRT 对象的 COM 对象,这是坏的)。你可以做的是这样的:
void __stdcall Direct3DSurfaceAccess(
IUnknown* surface, // pass a raw COM object (here I use IUnknown)
char* message,
size_t messageLength
) {
try {
// map to a C++/WinRT object here
auto d3dSurface = convert_from_abi<winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DSurface>(surface);
winrt::com_ptr<::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess> dxgiInterfaceAccess;
if (!d3dSurface.try_as(dxgiInterfaceAccess)) {
::sprintf_s(message, messageLength, "d3dSurface->try_as failed.");
return;
}
// Call other d3dSurface method as a test.
auto desc = d3dSurface.Description();
convert_from_abi
来自这里convert_from_abi function
在这种情况下,您可以像这样定义互操作函数:
[DllImport(@"Direct3DSurfaceAccess.dll")]
private static extern void Direct3DSurfaceAccess([MarshalAs(UnmanagedType.IUnknown)] object d3dSurface);
我正在为 Micorsoft HoloLens 2 创建 Unity 应用。
该应用使用 Windows.Media.Capture.MediaCapture 和 Microsoft.MixedReality.WebRTC.[=13= 捕捉和分享相机视频帧]
我在 MediaFrameReader.FrameArrived[= 上获得了由“NV12”格式化的 VideoMediaFrame.Direct3DSurface(Windows.Graphics.DirectX.Direct3D11.IDirect3DSurface) 42=]事件。
但我不知道如何从 IDirect3DSurface 对象获取像素数据。
我在下面试过:
C#代码
[DllImport("Direct3DSurfaceAccess")]
private static extern void Direct3DSurfaceAccess([MarshalAs(UnmanagedType.Interface)]Windows.Graphics.DirectX.Direct3D11.IDirect3DSurface d3dSurface);
// MediaFrameReader.FrameArrived event.
private void Nv12FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args) {
using var mediaFrameReference = sender.TryAcquireLatestFrame();
// I could't find how to get pixel data from IDirect3DSurface using C# only.
// Use C++/WinRT for access IDirect3DSurface as below.
Direct3DSurfaceAccess(mediaFrameReference.VideoMediaFrame.Direct3DSurface);
}
C++代码
void __stdcall Direct3DSurfaceAccess(const winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DSurface* d3dSurface) {
// Here, confirmed that d3dSurface is not null.
try {
// Issue is here.
winrt::com_ptr<::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess> dxgiInterfaceAccess;
d3dSurface->try_as(dxgiInterfaceAccess);
// App crashes on d3dSurface->try_as.
// Call other d3dSurface method as a test.
winrt::Windows::Graphics::DirectX::Direct3D11::Direct3DSurfaceDescription desc = d3dSurface->Description();
// Same above. App crashes on d3dSurface->Description.
// Unreachable to here...
// When succeeded to get the IDirect3DDxgiInterfaceAccess, read pixel data by below code.
winrt::com_ptr<::IDXGISurface> nativeSurface;
dxgiInterfaceAccess->GetInterface(IID_PPV_ARGS(nativeSurface.put()));
::DXGI_MAPPED_RECT rect;
nativeSurface->Map(&rect, DXGI_MAP_READ);
const BYTE* pixels = rect.pBits; // Read pixel data from the pointer.
nativeSurface->Unmap();
} catch (...) {
// Can't catch any exceptions.
// App freezes and downs on HoloLens 2.
}
}
我认为如何编组是错误的 IDirect3DSurface。
如何从 IDirect3DSurface 获取像素数据?
您不能在 dll 边界定义 C++/WinRT 对象指针,因为在二进制级别,它不是 COM 对象(因此 C# 正在传递映射到 C++/WinRT 对象的 COM 对象,这是坏的)。你可以做的是这样的:
void __stdcall Direct3DSurfaceAccess(
IUnknown* surface, // pass a raw COM object (here I use IUnknown)
char* message,
size_t messageLength
) {
try {
// map to a C++/WinRT object here
auto d3dSurface = convert_from_abi<winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DSurface>(surface);
winrt::com_ptr<::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess> dxgiInterfaceAccess;
if (!d3dSurface.try_as(dxgiInterfaceAccess)) {
::sprintf_s(message, messageLength, "d3dSurface->try_as failed.");
return;
}
// Call other d3dSurface method as a test.
auto desc = d3dSurface.Description();
convert_from_abi
来自这里convert_from_abi function
在这种情况下,您可以像这样定义互操作函数:
[DllImport(@"Direct3DSurfaceAccess.dll")]
private static extern void Direct3DSurfaceAccess([MarshalAs(UnmanagedType.IUnknown)] object d3dSurface);