MF SinkWriter 写入示例失败
MF SinkWriter Write Sample Failed
我正在尝试使用 MediaFoundation 将 ID3D11Texture2D 编码为 mp4。下面是我当前的代码。
正在初始化接收器编写器
private int InitializeSinkWriter(String outputFile, int videoWidth, int videoHeight)
{
IMFMediaType mediaTypeIn = null;
IMFMediaType mediaTypeOut = null;
IMFAttributes attributes = null;
int hr = 0;
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateAttributes(out attributes, 1);
if (Succeeded(hr)) hr = (int)attributes.SetUINT32(MFAttributesClsid.MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, 1);
if (Succeeded(hr)) hr = (int)attributes.SetUINT32(MFAttributesClsid.MF_LOW_LATENCY, 1);
// Create the sink writer
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateSinkWriterFromURL(outputFile, null, attributes, out sinkWriter);
// Create the output type
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateMediaType(out mediaTypeOut);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetGUID(MFAttributesClsid.MF_MT_MAJOR_TYPE, MFMediaType.Video);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetGUID(MFAttributesClsid.MF_TRANSCODE_CONTAINERTYPE, MFTranscodeContainerType.MPEG4);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetGUID(MFAttributesClsid.MF_MT_SUBTYPE, MFMediaType.H264);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetUINT32(MFAttributesClsid.MF_MT_AVG_BITRATE, videoBitRate);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetUINT32(MFAttributesClsid.MF_MT_INTERLACE_MODE, (int)MFVideoInterlaceMode.Progressive);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeSize(mediaTypeOut, MFAttributesClsid.MF_MT_FRAME_SIZE, videoWidth, videoHeight);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeRatio(mediaTypeOut, MFAttributesClsid.MF_MT_FRAME_RATE, VIDEO_FPS, 1);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeRatio(mediaTypeOut, MFAttributesClsid.MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
if (Succeeded(hr)) hr = (int)sinkWriter.AddStream(mediaTypeOut, out streamIndex);
// Create the input type
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateMediaType(out mediaTypeIn);
if (Succeeded(hr)) hr = (int)mediaTypeIn.SetGUID(MFAttributesClsid.MF_MT_MAJOR_TYPE, MFMediaType.Video);
if (Succeeded(hr)) hr = (int)mediaTypeIn.SetGUID(MFAttributesClsid.MF_MT_SUBTYPE, MFMediaType.ARGB32);
if (Succeeded(hr)) hr = (int)mediaTypeIn.SetUINT32(MFAttributesClsid.MF_SA_D3D11_AWARE, 1);
if (Succeeded(hr)) hr = (int)mediaTypeIn.SetUINT32(MFAttributesClsid.MF_MT_INTERLACE_MODE, (int)MFVideoInterlaceMode.Progressive);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeSize(mediaTypeIn, MFAttributesClsid.MF_MT_FRAME_SIZE, videoWidth, videoHeight);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeRatio(mediaTypeIn, MFAttributesClsid.MF_MT_FRAME_RATE, VIDEO_FPS, 1);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeRatio(mediaTypeIn, MFAttributesClsid.MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
if (Succeeded(hr)) hr = (int)sinkWriter.SetInputMediaType(streamIndex, mediaTypeIn, null);
// Start accepting data
if (Succeeded(hr)) hr = (int)sinkWriter.BeginWriting();
COMBase.SafeRelease(mediaTypeOut);
COMBase.SafeRelease(mediaTypeIn);
return hr;
}
写作框架
int hr = 0;
IMFSample sample = null;
IMFMediaBuffer buffer = null;
IMF2DBuffer p2Dbuffer = null;
object texNativeObject = Marshal.GetObjectForIUnknown(surface.NativePointer);
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateDXGISurfaceBuffer(new Guid("6f15aaf2-d208-4e89-9ab4-489535d34f9c"), texNativeObject, 0, false, out p2Dbuffer);
buffer = MFVideoEncoderST.ReinterpretCast<IMF2DBuffer,IMFMediaBuffer>(p2Dbuffer);
int length=0;
if (Succeeded(hr)) hr = (int)p2Dbuffer.GetContiguousLength(out length);
if (Succeeded(hr)) hr = (int)buffer.SetCurrentLength(length);
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateVideoSampleFromSurface(null, out sample);
if (Succeeded(hr)) hr = (int)sample.AddBuffer(buffer);
if (Succeeded(hr)) hr = (int)sample.SetSampleTime(prevRecordingDuration);
if (Succeeded(hr)) hr = (int)sample.SetSampleDuration((recordDuration - prevRecordingDuration));
if (Succeeded(hr)) hr = (int)sinkWriter.WriteSample(streamIndex, sample);
COMBase.SafeRelease(sample);
COMBase.SafeRelease(buffer);
使用 MFTRACE 时出现以下错误。
02:48:04.99463 CMFSinkWriterDetours::WriteSample @024BEA18 Stream Index 0x0, Sample @17CEACE0, Time 571ms, Duration 16ms, Buffers 1, Size 4196352B,2088,2008 02:48:04.99465 CMFSinkWriterDetours::WriteSample @024BEA18 failed hr=0x887A0005 (null)2088,2008
02:48:05.01090 CMFSinkWriterDetours::WriteSample @024BEA18 Stream Index 0x0, Sample @17CE9FC0, Time 587ms, Duration 17ms, Buffers 1, Size 4196352B,2088,2008 02:48:05.01091 CMFSinkWriterDetours::WriteSample @024BEA18 failed hr=0x887A0005 (null)2088,2008
02:48:05.02712 CMFSinkWriterDetours::WriteSample @024BEA18 Stream Index 0x0, Sample @17CEACE0, Time 604ms, Duration 16ms, Buffers 1, Size 4196352B,2088,2008 02:48:05.02713 CMFSinkWriterDetours::WriteSample @024BEA18 failed hr=0x887A0005 (null)
谁能告诉我我的代码有什么问题吗？我只能生成0字节的mp4文件。
我在这里遇到了一些潜在的问题。 Roman 提到了两个大问题，所以我将详细说明。我还有一些其他的批评/建议给你。
不使用IMFDXGIDeviceManager
为了在 Media Foundation 中使用硬件加速，您需要创建一个 DirectX 设备管理器对象，可以是 IDirect3DDeviceManager9 for DX9 or in your case an IMFDXGIDeviceManager for DXGI. I strongly suggest reading all the MSDN documentation of that interface. The reason this is necessary is because the same DX device must be shared across all the cooperating hardware MF transforms being used, since they all need access to the shared GPU memory the device controls, and each one needs exclusive control of the device while it's working, so a locking system is needed. The device manager object provides that locking system, and is also the standard way of providing a DX device to one or more transforms. For DXGI, you create this using MFCreateDXGIDeviceManager。
从那里，您需要创建您的 DX11 设备，并使用您的 DX11 设备调用 IMFDXGIDeviceManager::ResetDevice。然后您需要为 Sink Writer 本身设置设备管理器，这在您上面提供的代码中没有完成。这是这样完成的：
// ... inside your InitializeSinkWriter function that you listed above
// I'm assuming you've already created and set up the DXGI device manager elsewhere
IMFDXGIDeviceManager pDeviceManager;
// Passing 3 as the argument because we're adding 3 attributes immediately, saves re-allocations
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateAttributes(out attributes, 3);
if (Succeeded(hr)) hr = (int)attributes.SetUINT32(MFAttributesClsid.MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, 1);
if (Succeeded(hr)) hr = (int)attributes.SetUINT32(MFAttributesClsid.MF_LOW_LATENCY, 1);
// Here's the key piece!
if (Succeeded(hr)) hr = (int)attributes.SetUnknown(MFAttributesClsid.MF_SINK_WRITER_D3D_MANAGER, pDeviceManager);
// Create the sink writer
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateSinkWriterFromURL(outputFile, null, attributes, out sinkWriter);
这实际上会为硬件编码器启用 D3D11 支持，并允许它读取您传入的 Texture2D。值得注意的是 MF_SINK_WRITER_D3D_MANAGER 适用于 DX9 和 DXGI 设备管理器.
编码器缓冲同一纹理的多个 IMFSample 个实例
这也是您问题的一个潜在原因 - 至少它会导致很多意外行为，即使它不是明显问题的原因。基于 Roman 的评论，许多编码器将缓冲多个帧作为其编码过程的一部分。使用 Sink Writer 时您不会看到该行为，因为它会为您处理所有细节工作。但是，您试图完成的事情（即发送 D3D11 纹理作为输入帧）级别足够低，您开始不得不担心 Sink Writer 使用的编码器 MFT 的内部细节。
大多数视频编码器 MFT 将使用一定大小的内部缓冲区来存储通过 IMFTransform::ProcessInput. This has the side effect that multiple samples must be provided as inputs before any output will be generated. Video encoders need access to multiple samples in order because they use the subsequent frames to determine how to encode the current frame. In other words, if the decoder is working on frame 0, it might need to see frames 1, 2, and 3 as well. From a technical standpoint, this is because of things like inter-frame prediction and motion estimation. Once the encoder is finished processing the oldest sample, it generates an output buffer (another IMFSample object, but this time on the output side via IMFTransform::ProcessOutput) then discards the input sample it was working on (by calling IUnknown::Release), then requests more input, and eventually moves on to the next frame. You can read more about this process in the MSDN article Processing Data in the Encoder
提供的最后 N 个样本
正如 Roman 所暗示的，这意味着您将 ID3D11Texture2D 封装在 IMFSample 中的 IMFMediaBuffer 中，然后将其传递给 Sink Writer。作为编码过程的一部分，该样本很可能被编码器缓冲。由于编码器正在工作，Texture2D 的内容可能会发生变化，这可能会导致各种问题。即使这不会导致程序错误，也肯定会导致非常奇怪的编码视频输出。想象一下，如果编码器试图预测一帧的视觉内容在下一帧中如何变化，然后两帧的实际视觉内容都从编码器下更新出来！
这个特定问题的发生是因为编码器只有一个指向您的 IMFSample 实例的指针引用，它最终只是指向您的 ID3D11Texture2D 对象的指针本身，并且该对象是一种指向可变图形内存的指针引用。最终，由于程序的某些其他部分，该图形内存的内容正在发生变化，但由于它总是更新相同的 GPU 纹理，因此您发送编码器的每个样本都指向相同的单个纹理。这意味着每当您更新纹理时，通过更改 GPU 内存，所有活动的 IMFSample 对象都将反映这些更改，因为它们都有效地指向相同的 GPU 纹理。
要解决此问题，您需要分配多个 ID3D11Texture2D 对象，以便在将其提供给 Sink Writer 时可以将一个纹理与一个 IMFSample 配对。这将通过使每个样本指向一个唯一的纹理来解决所有样本指向同一个 GPU 纹理的问题。但是，您不一定知道需要创建多少纹理，因此处理此问题的最安全方法是编写您自己的纹理分配器。这仍然可以在 C# 中完成，这是值得的，MediaFoundation.NET 定义了您需要使用的接口。
分配器应维护一个 "free" SharpDX.Texture2D 对象的列表 - 那些当前未被接收器写入器/编码器使用的对象。您的程序应该能够从分配器请求新的纹理对象，在这种情况下，它将 return 来自空闲列表的对象，或者创建一个新的纹理来满足请求。
下一个问题是知道 IMFSample 对象何时被编码器丢弃，以便您可以将附加的纹理添加回空闲列表。碰巧的是，您当前使用的 MFCreateVideoSampleFromSurface 函数分配实现 IMFTrackedSample 接口的样本。您将需要该接口，以便在释放示例时收到通知，以便您可以回收 Texture2D 个对象。
诀窍是你必须告诉样本你是分配者。首先，您的分配器 class 需要在编码器释放样本时实现作为参数传递的 IMFAsyncCallback. If you set your allocator class on the sample via IMFTrackedSample::SetAllocator, your allocator's IMFAsyncCallback::Invoke method will be called, with an IMFAsyncResult。这是分配器 class 的一般示例。
sealed class TextureAllocator : IMFAsyncCallback, IDisposable
{
private ConcurrentStack<SharpDX.Direct3D11.Texture2D> m_freeStack;
private static readonly Guid s_IID_ID3D11Texture2D = new Guid("6f15aaf2-d208-4e89-9ab4-489535d34f9c");
// If all textures are the exact same size and color format,
// consider making those parameters private class members and
// requiring they be specified as arguments to the constructor.
public TextureAllocator()
{
m_freeStack = new ConcurrentStack<SharpDX.Direct3D11.Texture2D>();
}
private bool disposedValue = false;
private void Dispose(bool disposing)
{
if(!disposedValue)
{
if(disposing)
{
// Dispose managed resources here
}
if(m_freeStack != null)
{
SharpDX.Direct3D11.Texture2D texture;
while(m_freeStack.TryPop(out texture))
{
texture.Dispose();
}
m_freeStack = null;
}
disposedValue = true;
}
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
~TextureAllocator()
{
Dispose(false);
}
private SharpDX.Direct3D11.Texture2D InternalAllocateNewTexture()
{
// Allocate a new texture with your format, size, etc here.
}
public SharpDX.Direct3D11.Texture2D AllocateTexture()
{
SharpDX.Direct3D11.Texture2D existingTexture;
if(m_freeStack.TryPop(out existingTexture))
{
return existingTexture;
}
else
{
return InternalAllocateNewTexture();
}
}
public IMFSample CreateSampleAndAllocateTexture()
{
IMFSample pSample;
IMFTrackedSample pTrackedSample;
HResult hr;
// Create the video sample. This function returns an IMFTrackedSample per MSDN
hr = MFExtern.MFCreateVideoSampleFromSurface(null, out pSample);
MFError.ThrowExceptionForHR(hr);
// Query the IMFSample to see if it implements IMFTrackedSample
pTrackedSample = pSample as IMFTrackedSample;
if(pTrackedSample == null)
{
// Throw an exception if we didn't get an IMFTrackedSample
// but this shouldn't happen in practice.
throw new InvalidCastException("MFCreateVideoSampleFromSurface returned a sample that did not implement IMFTrackedSample");
}
// Use our own class to allocate a texture
SharpDX.Direct3D11.Texture2D availableTexture = AllocateTexture();
// Convert the texture's native ID3D11Texture2D pointer into
// an IUnknown (represented as as System.Object)
object texNativeObject = Marshal.GetObjectForIUnknown(availableTexture.NativePointer);
// Create the media buffer from the texture
IMFMediaBuffer p2DBuffer;
hr = MFExtern.MFCreateDXGISurfaceBuffer(s_IID_ID3D11Texture2D, texNativeObject, 0, false, out p2DBuffer);
// Release the object-as-IUnknown we created above
COMBase.SafeRelease(texNativeObject);
// If media buffer creation failed, throw an exception
MFError.ThrowExceptionForHR(hr);
// Set the owning instance of this class as the allocator
// for IMFTrackedSample to notify when the sample is released
pTrackedSample.SetAllocator(this, null);
// Attach the created buffer to the sample
pTrackedSample.AddBuffer(p2DBuffer);
return pTrackedSample;
}
// This is public so any textures you allocate but don't make IMFSamples
// out of can be returned to the allocator manually.
public void ReturnFreeTexture(SharpDX.Direct3D11.Texture2D freeTexture)
{
m_freeStack.Push(freeTexture);
}
// IMFAsyncCallback.GetParameters
// This is allowed to return E_NOTIMPL as a way of specifying
// there are no special parameters.
public HResult GetParameters(out MFAsync pdwFlags, out MFAsyncCallbackQueue pdwQueue)
{
pdwFlags = MFAsync.None;
pdwQueue = MFAsyncCallbackQueue.Standard;
return HResult.E_NOTIMPL;
}
public HResult Invoke(IMFAsyncResult pResult)
{
object pUnkObject;
IMFSample pSample = null;
IMFMediaBuffer pBuffer = null;
IMFDXGIBuffer pDXGIBuffer = null;
// Get the IUnknown out of the IMFAsyncResult if there is one
HResult hr = pResult.GetObject(out pUnkObject);
if(Succeeded(hr))
{
pSample = pUnkObject as IMFSample;
}
if(pSample != null)
{
// Based on your implementation, there should only be one
// buffer attached to one sample, so we can always grab the
// first buffer. You could add some error checking here to make
// sure the sample has a buffer count that is 1.
hr = pSample.GetBufferByIndex(0, out pBuffer);
}
if(Succeeded(hr))
{
// Query the IMFMediaBuffer to see if it implements IMFDXGIBuffer
pDXGIBuffer = pBuffer as IMFDXGIBuffer;
}
if(pDXGIBuffer != null)
{
// Got an IMFDXGIBuffer, so we can extract the internal
// ID3D11Texture2D and make a new SharpDX.Texture2D wrapper.
hr = pDXGIBuffer.GetResource(s_IID_ID3D11Texture2D, out pUnkObject);
}
if(Succeeded(hr))
{
// If we got here, pUnkObject is the native D3D11 Texture2D as
// a System.Object, but it's unlikely you have an interface
// definition for ID3D11Texture2D handy, so we can't just cast
// the object to the proper interface.
// Happily, SharpDX supports wrapping System.Object within
// SharpDX.ComObject which makes things pretty easy.
SharpDX.ComObject comWrapper = new SharpDX.ComObject(pUnkObject);
// If this doesn't work, or you're using something like SlimDX
// which doesn't support object wrapping the same way, the below
// code is an alternative way.
/*
IntPtr pD3DTexture2D = Marshal.GetIUnknownForObject(pUnkObject);
// Create your wrapper object here, like this for SharpDX
SharpDX.ComObject comWrapper = new SharpDX.ComObject(pD3DTexture2D);
// or like this for SlimDX
SlimDX.Direct3D11.Texture2D.FromPointer(pD3DTexture2D);
Marshal.Release(pD3DTexture2D);
*/
// You might need to query comWrapper for a SharpDX.DXGI.Resource
// first, then query that for the SharpDX.Direct3D11.Texture2D.
SharpDX.Direct3D11.Texture2D texture = comWrapper.QueryInterface<SharpDX.Direct3D11.Texture2D>();
if(texture != null)
{
// Now you can add "texture" back to the allocator's free list
ReturnFreeTexture(texture);
}
}
}
}
在 Sink Writer 输入媒体类型上设置 MF_SA_D3D_AWARE
我不认为这会导致 HRESULT 您遇到的问题，但无论如何这都不是正确的做法。 MF_SA_D3D_AWARE（及其对应的 DX11，MF_SA_D3D11_AWARE）是由 IMFTransform 对象设置的属性，用于通知您转换分别通过 DX9 或 DX11 支持图形加速。无需在 Sink Writer 的输入媒体类型上进行设置。
在 texNativeObject
上没有 SafeRelease
我建议在 texNativeObject 上调用 COMBase.SafeRelease()，否则您可能会泄漏内存。那，或者你会不必要地延长那个 COM 对象的生命周期，直到 GC 为你清理引用计数
不必要的转换
这是您上面代码的一部分：
buffer = MFVideoEncoderST.ReinterpretCast<IMF2DBuffer,IMFMediaBuffer>(p2Dbuffer);
int length=0;
if (Succeeded(hr)) hr = (int)p2Dbuffer.GetContiguousLength(out length);
if (Succeeded(hr)) hr = (int)buffer.SetCurrentLength(length);
我不确定您的 ReinterpretCast 函数在做什么，但是如果您 确实 需要在 C# 中执行 QueryInterface 样式转换，您可以只使用 as 运算符或常规转换。
// pMediaBuffer is of type IMFMediaBuffer and has been created elsewhere
IMF2DBuffer p2DBuffer = pMediaBuffer as IMF2DBuffer;
if(p2DBuffer != null)
{
// pMediaBuffer is an IMFMediaBuffer that also implements IMF2DBuffer
}
else
{
// pMediaBuffer does not implement IMF2DBuffer
}
第一期：IMFDXGIDeviceManager::ResetDevice总是失败。
在我之前回答的评论中与@kripto 合作后，我们诊断出了许多其他问题。最大的问题是设置 IMFDXGIDeviceManager in order to enable a hardware H.264 encoder MFT to accept Direct3D11 Texture2D samples, contained inside an IMFDXGIBuffer。代码中有一个很难发现的错误：
// pDevice is a SharpDX.Direct3D11.Texture2D instance
// pDevice.NativePointer is an IntPtr that refers to the native IDirect3D11Device
// being wrapped by SharpDX.
IMFDXGIDeviceManager pDeviceManager;
object d3dDevice = Marshal.GetIUnknownForObject(pDevice.NativePointer);
HResult hr = MFExtern.MFCreateDXGIDeviceManager(out resetToken, out pDeviceManager);
if(Succeeded(hr))
{
// The signature of this is:
// HResult ResetDevice(object d3d11device, int resetToken);
hr = pDeviceManager.ResetDevice(d3dDevice, resetToken);
}
上面的代码是这样的。设备管理器已创建，但为了让编码器 MFT 访问 Texture2D 个样本，它需要创建纹理的同一 Direct3D 设备的副本。因此，您必须在设备管理器上调用 IMFDXGIDeviceManager::ResetDevice 才能为其提供 Direct3D 设备。有关 ResetDevice 的一些重要脚注，请参阅 [1]。 SharpDX 仅提供对指向本机 IDirect3D11Device 的 IntPtr 的访问，但 MediaFoundation.NET 接口需要传入 object 来代替。
看到错误了吗？上面的代码类型检查和编译都很好，但包含一个严重错误。错误是使用 Marshal.GetIUnknownForObject instead of Marshal.GetObjectForIUnknown。有趣的是，因为 object 可以很好地装箱 IntPtr，所以您可以使用完全相反的编组函数，它仍然可以很好地编译。问题是我们正在尝试将 IntPtr 转换为 object 内的 .NET RCW，这是 MediaFoundation.NET 中的 ResetDevice 所期望的。此错误导致 ResetDevice 到 return E_INVALIDARG 而不是正常工作。
第二期：奇怪的编码器输出
第二个问题是 Intel Quick Sync Video H.264 Encoder MFT 不是特别满意，虽然它被正确创建，但在结果文件的开头有一两秒的黑色输出，以及前几秒的阻塞和运动错误，有时视频的一半是灰色的，没有显示实际复制的桌面图像。
我想确保实际的 Texture2D 对象被正确发送到编码器，所以我写了一个简单的 class 将 Direct3D 11 Texture2D 转储到 .png 文件。我在这里为任何需要它的人提供了它——这需要 SharpDX 和 MediaFoundation.NET 才能工作，尽管您可以使用 CopyMemory in a loop to account for the different strides. Note that this is only set up to work with textures in DXGI.Format.B8G8R8A8_UNorm 格式删除 MF 依赖。它可能适用于其他格式的纹理，但输出看起来很奇怪。
using System;
using System.Drawing;
namespace ScreenCapture
{
class Texture2DDownload : IDisposable
{
private SharpDX.Direct3D11.Device m_pDevice;
private SharpDX.Direct3D11.Texture2D m_pDebugTexture;
public Texture2DDownload(SharpDX.Direct3D11.Device pDevice)
{
m_pDevice = pDevice;
}
/// <summary>
/// Compare all the relevant properties of the texture descriptions for both input textures.
/// </summary>
/// <param name="texSource">The source texture</param>
/// <param name="texDest">The destination texture that will have the source data copied into it</param>
/// <returns>true if the source texture can be copied to the destination, false if their descriptions are incompatible</returns>
public static bool TextureCanBeCopied(SharpDX.Direct3D11.Texture2D texSource, SharpDX.Direct3D11.Texture2D texDest)
{
if(texSource.Description.ArraySize != texDest.Description.ArraySize)
return false;
if(texSource.Description.Format != texDest.Description.Format)
return false;
if(texSource.Description.Height != texDest.Description.Height)
return false;
if(texSource.Description.MipLevels != texDest.Description.MipLevels)
return false;
if(texSource.Description.SampleDescription.Count != texDest.Description.SampleDescription.Count)
return false;
if(texSource.Description.SampleDescription.Quality != texDest.Description.SampleDescription.Quality)
return false;
if(texSource.Description.Width != texDest.Description.Width)
return false;
return true;
}
/// <summary>
/// Saves the contents of a <see cref="SharpDX.Direct3D11.Texture2D"/> to a file with name contained in <paramref name="filename"/> using the specified <see cref="System.Drawing.Imaging.ImageFormat"/>.
/// </summary>
/// <param name="texture">The <see cref="SharpDX.Direct3D11.Texture2D"/> containing the data to save.</param>
/// <param name="filename">The filename on disk where the output image should be saved.</param>
/// <param name="imageFormat">The format to use when saving the output file.</param>
public void SaveTextureToFile(SharpDX.Direct3D11.Texture2D texture, string filename, System.Drawing.Imaging.ImageFormat imageFormat)
{
// If the existing debug texture doesn't exist, or the incoming texture is different than the existing debug texture...
if(m_pDebugTexture == null || !TextureCanBeCopied(m_pDebugTexture, texture))
{
// Dispose of any existing texture
if(m_pDebugTexture != null)
{
m_pDebugTexture.Dispose();
}
// Copy the original texture's description...
SharpDX.Direct3D11.Texture2DDescription newDescription = texture.Description;
// Then modify the parameters to create a CPU-readable staging texture
newDescription.BindFlags = SharpDX.Direct3D11.BindFlags.None;
newDescription.CpuAccessFlags = SharpDX.Direct3D11.CpuAccessFlags.Read;
newDescription.OptionFlags = SharpDX.Direct3D11.ResourceOptionFlags.None;
newDescription.Usage = SharpDX.Direct3D11.ResourceUsage.Staging;
// Re-generate the debug texture by copying the new texture's description
m_pDebugTexture = new SharpDX.Direct3D11.Texture2D(m_pDevice, newDescription);
}
// Copy the texture to our debug texture
m_pDevice.ImmediateContext.CopyResource(texture, m_pDebugTexture);
// Map the debug texture's resource 0 for read mode
SharpDX.DataStream data;
SharpDX.DataBox dbox = m_pDevice.ImmediateContext.MapSubresource(m_pDebugTexture, 0, 0, SharpDX.Direct3D11.MapMode.Read, SharpDX.Direct3D11.MapFlags.None, out data);
// Create a bitmap that's the same size as the debug texture
Bitmap b = new Bitmap(m_pDebugTexture.Description.Width, m_pDebugTexture.Description.Height, System.Drawing.Imaging.PixelFormat.Format32bppRgb);
// Lock the bitmap data to get access to the native bitmap pointer
System.Drawing.Imaging.BitmapData bd = b.LockBits(new Rectangle(0, 0, b.Width, b.Height), System.Drawing.Imaging.ImageLockMode.WriteOnly, System.Drawing.Imaging.PixelFormat.Format32bppRgb);
// Use the native pointers to do a native-to-native memory copy from the mapped subresource to the bitmap data
// WARNING: This might totally blow up if you're using a different color format than B8G8R8A8_UNorm, I don't know how planar formats are structured as D3D textures!
//
// You can use Win32 CopyMemory to do the below copy if need be, but you have to do it in a loop to respect the Stride and RowPitch parameters in case the texture width
// isn't on an aligned byte boundary.
MediaFoundation.MFExtern.MFCopyImage(bd.Scan0, bd.Stride, dbox.DataPointer, dbox.RowPitch, bd.Width * 4, bd.Height);
/// Unlock the bitmap
b.UnlockBits(bd);
// Unmap the subresource mapping, ignore the SharpDX.DataStream because we don't need it.
m_pDevice.ImmediateContext.UnmapSubresource(m_pDebugTexture, 0);
data = null;
// Save the bitmap to the desired filename
b.Save(filename, imageFormat);
b.Dispose();
b = null;
}
#region IDisposable Support
private bool disposedValue = false; // To detect redundant calls
protected virtual void Dispose(bool disposing)
{
if(!disposedValue)
{
if(disposing)
{
}
if(m_pDebugTexture != null)
{
m_pDebugTexture.Dispose();
}
disposedValue = true;
}
}
// TODO: override a finalizer only if Dispose(bool disposing) above has code to free unmanaged resources.
~Texture2DDownload() {
// Do not change this code. Put cleanup code in Dispose(bool disposing) above.
Dispose(false);
}
// This code added to correctly implement the disposable pattern.
public void Dispose()
{
// Do not change this code. Put cleanup code in Dispose(bool disposing) above.
Dispose(true);
GC.SuppressFinalize(this);
}
#endregion
}
}
一旦我确认我有好的图像进入编码器，我发现代码没有调用 IMFSinkWriter::SendStreamTick after calling IMFSinkWriter::BeginWriting 而是在发送第一个 IMFSample 之前。初始样本也有一个非零的时间增量，这导致了初始黑色输出。为了解决这个问题，我添加了以下代码：
// Existing code to set the sample time and duration
// recordDuration is the current frame time in 100-nanosecond units
// prevRecordingDuration is the frame time of the last frame in
// 100-nanosecond units
sample.SetSampleTime(recordDuration);
sample.SetSampleDuration(recordDuration - prevRecordingDuration);
// The fix is here:
if(frames == 0)
{
sinkWriter.SendStreamTick(streamIndex, recordDuration);
sample.SetUINT32(MFAttributesClsid.MFSampleExtension_Discontinuity, 1);
}
sinkWriter.WriteSample(streamIndex, sample);
frames++;
通过向接收器编写器发送一个流标记，它确定 recordDuration 中的任何值现在都被视为输出视频流的时间 = 0 点。换句话说，一旦您调用 SetStreamTick 并传入帧时间戳，所有后续时间戳都会从中减去该初始时间戳。这就是让第一个样本帧立即显示在输出文件中的方法。
此外，无论何时调用SendStreamTick，之后直接提供给sink writer的样本必须在其属性上将MFSampleExtension_Discontinuity设置为1。这意味着发送的样本中存在间隙，并且传递给编码器的帧是该间隙之后的第一帧。这或多或少地告诉编码器从样本中制作一个关键帧，这可以防止我之前看到的运动和阻塞效果。
结果
实施这些修复后，我测试了应用程序并实现了 1920x1080 分辨率和每秒 60 帧输出的全屏捕获。比特率设置为 4096 kbit。 CPU 在英特尔 i7-4510U 笔记本电脑 CPU 上，大多数工作负载的使用率在 2.5% 到 7% 之间 - 极端的运动会使其达到约 10%。通过 SysInternals 的 Process Explorer 获得的 GPU 利用率在 1% 到 2% 之间。
[1] 我相信其中一些是 Direct3D 9 的遗留问题，当时 DirectX API 中没有很好地内置多线程支持，并且每当任何组件使用时都必须独占锁定设备它（即解码器、渲染器、编码器）。使用 D3D 9 你可以调用 ResetDevice 但之后就再也不能使用你自己的设备指针了。相反，您甚至必须在自己的代码中调用 LockDevice 和 UnlockDevice 来获取设备指针，因为 MFT 可能在同一时刻使用该设备。在 Direct3D 11 中，在 MFT 和控制应用程序中同时使用同一设备似乎没有问题 - 尽管如果发生任何随机崩溃，我建议阅读大量有关如何 IMFDXGIDeviceManager::LockDevice 和 UnlockDevice 工作并实施这些以确保设备始终受到完全控制。
我正在尝试使用 MediaFoundation 将 ID3D11Texture2D 编码为 mp4。下面是我当前的代码。
正在初始化接收器编写器
private int InitializeSinkWriter(String outputFile, int videoWidth, int videoHeight)
{
IMFMediaType mediaTypeIn = null;
IMFMediaType mediaTypeOut = null;
IMFAttributes attributes = null;
int hr = 0;
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateAttributes(out attributes, 1);
if (Succeeded(hr)) hr = (int)attributes.SetUINT32(MFAttributesClsid.MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, 1);
if (Succeeded(hr)) hr = (int)attributes.SetUINT32(MFAttributesClsid.MF_LOW_LATENCY, 1);
// Create the sink writer
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateSinkWriterFromURL(outputFile, null, attributes, out sinkWriter);
// Create the output type
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateMediaType(out mediaTypeOut);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetGUID(MFAttributesClsid.MF_MT_MAJOR_TYPE, MFMediaType.Video);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetGUID(MFAttributesClsid.MF_TRANSCODE_CONTAINERTYPE, MFTranscodeContainerType.MPEG4);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetGUID(MFAttributesClsid.MF_MT_SUBTYPE, MFMediaType.H264);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetUINT32(MFAttributesClsid.MF_MT_AVG_BITRATE, videoBitRate);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetUINT32(MFAttributesClsid.MF_MT_INTERLACE_MODE, (int)MFVideoInterlaceMode.Progressive);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeSize(mediaTypeOut, MFAttributesClsid.MF_MT_FRAME_SIZE, videoWidth, videoHeight);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeRatio(mediaTypeOut, MFAttributesClsid.MF_MT_FRAME_RATE, VIDEO_FPS, 1);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeRatio(mediaTypeOut, MFAttributesClsid.MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
if (Succeeded(hr)) hr = (int)sinkWriter.AddStream(mediaTypeOut, out streamIndex);
// Create the input type
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateMediaType(out mediaTypeIn);
if (Succeeded(hr)) hr = (int)mediaTypeIn.SetGUID(MFAttributesClsid.MF_MT_MAJOR_TYPE, MFMediaType.Video);
if (Succeeded(hr)) hr = (int)mediaTypeIn.SetGUID(MFAttributesClsid.MF_MT_SUBTYPE, MFMediaType.ARGB32);
if (Succeeded(hr)) hr = (int)mediaTypeIn.SetUINT32(MFAttributesClsid.MF_SA_D3D11_AWARE, 1);
if (Succeeded(hr)) hr = (int)mediaTypeIn.SetUINT32(MFAttributesClsid.MF_MT_INTERLACE_MODE, (int)MFVideoInterlaceMode.Progressive);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeSize(mediaTypeIn, MFAttributesClsid.MF_MT_FRAME_SIZE, videoWidth, videoHeight);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeRatio(mediaTypeIn, MFAttributesClsid.MF_MT_FRAME_RATE, VIDEO_FPS, 1);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeRatio(mediaTypeIn, MFAttributesClsid.MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
if (Succeeded(hr)) hr = (int)sinkWriter.SetInputMediaType(streamIndex, mediaTypeIn, null);
// Start accepting data
if (Succeeded(hr)) hr = (int)sinkWriter.BeginWriting();
COMBase.SafeRelease(mediaTypeOut);
COMBase.SafeRelease(mediaTypeIn);
return hr;
}
写作框架
int hr = 0;
IMFSample sample = null;
IMFMediaBuffer buffer = null;
IMF2DBuffer p2Dbuffer = null;
object texNativeObject = Marshal.GetObjectForIUnknown(surface.NativePointer);
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateDXGISurfaceBuffer(new Guid("6f15aaf2-d208-4e89-9ab4-489535d34f9c"), texNativeObject, 0, false, out p2Dbuffer);
buffer = MFVideoEncoderST.ReinterpretCast<IMF2DBuffer,IMFMediaBuffer>(p2Dbuffer);
int length=0;
if (Succeeded(hr)) hr = (int)p2Dbuffer.GetContiguousLength(out length);
if (Succeeded(hr)) hr = (int)buffer.SetCurrentLength(length);
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateVideoSampleFromSurface(null, out sample);
if (Succeeded(hr)) hr = (int)sample.AddBuffer(buffer);
if (Succeeded(hr)) hr = (int)sample.SetSampleTime(prevRecordingDuration);
if (Succeeded(hr)) hr = (int)sample.SetSampleDuration((recordDuration - prevRecordingDuration));
if (Succeeded(hr)) hr = (int)sinkWriter.WriteSample(streamIndex, sample);
COMBase.SafeRelease(sample);
COMBase.SafeRelease(buffer);
使用 MFTRACE 时出现以下错误。
02:48:04.99463 CMFSinkWriterDetours::WriteSample @024BEA18 Stream Index 0x0, Sample @17CEACE0, Time 571ms, Duration 16ms, Buffers 1, Size 4196352B,2088,2008 02:48:04.99465 CMFSinkWriterDetours::WriteSample @024BEA18 failed hr=0x887A0005 (null)2088,2008
02:48:05.01090 CMFSinkWriterDetours::WriteSample @024BEA18 Stream Index 0x0, Sample @17CE9FC0, Time 587ms, Duration 17ms, Buffers 1, Size 4196352B,2088,2008 02:48:05.01091 CMFSinkWriterDetours::WriteSample @024BEA18 failed hr=0x887A0005 (null)2088,2008
02:48:05.02712 CMFSinkWriterDetours::WriteSample @024BEA18 Stream Index 0x0, Sample @17CEACE0, Time 604ms, Duration 16ms, Buffers 1, Size 4196352B,2088,2008 02:48:05.02713 CMFSinkWriterDetours::WriteSample @024BEA18 failed hr=0x887A0005 (null)
谁能告诉我我的代码有什么问题吗？我只能生成0字节的mp4文件。
我在这里遇到了一些潜在的问题。 Roman 提到了两个大问题，所以我将详细说明。我还有一些其他的批评/建议给你。
不使用IMFDXGIDeviceManager
为了在 Media Foundation 中使用硬件加速，您需要创建一个 DirectX 设备管理器对象，可以是 IDirect3DDeviceManager9 for DX9 or in your case an IMFDXGIDeviceManager for DXGI. I strongly suggest reading all the MSDN documentation of that interface. The reason this is necessary is because the same DX device must be shared across all the cooperating hardware MF transforms being used, since they all need access to the shared GPU memory the device controls, and each one needs exclusive control of the device while it's working, so a locking system is needed. The device manager object provides that locking system, and is also the standard way of providing a DX device to one or more transforms. For DXGI, you create this using MFCreateDXGIDeviceManager。
从那里，您需要创建您的 DX11 设备，并使用您的 DX11 设备调用 IMFDXGIDeviceManager::ResetDevice。然后您需要为 Sink Writer 本身设置设备管理器，这在您上面提供的代码中没有完成。这是这样完成的：
// ... inside your InitializeSinkWriter function that you listed above
// I'm assuming you've already created and set up the DXGI device manager elsewhere
IMFDXGIDeviceManager pDeviceManager;
// Passing 3 as the argument because we're adding 3 attributes immediately, saves re-allocations
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateAttributes(out attributes, 3);
if (Succeeded(hr)) hr = (int)attributes.SetUINT32(MFAttributesClsid.MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, 1);
if (Succeeded(hr)) hr = (int)attributes.SetUINT32(MFAttributesClsid.MF_LOW_LATENCY, 1);
// Here's the key piece!
if (Succeeded(hr)) hr = (int)attributes.SetUnknown(MFAttributesClsid.MF_SINK_WRITER_D3D_MANAGER, pDeviceManager);
// Create the sink writer
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateSinkWriterFromURL(outputFile, null, attributes, out sinkWriter);
这实际上会为硬件编码器启用 D3D11 支持，并允许它读取您传入的 Texture2D。值得注意的是 MF_SINK_WRITER_D3D_MANAGER 适用于 DX9 和 DXGI 设备管理器.
编码器缓冲同一纹理的多个 IMFSample 个实例
这也是您问题的一个潜在原因 - 至少它会导致很多意外行为，即使它不是明显问题的原因。基于 Roman 的评论，许多编码器将缓冲多个帧作为其编码过程的一部分。使用 Sink Writer 时您不会看到该行为，因为它会为您处理所有细节工作。但是，您试图完成的事情（即发送 D3D11 纹理作为输入帧）级别足够低，您开始不得不担心 Sink Writer 使用的编码器 MFT 的内部细节。
大多数视频编码器 MFT 将使用一定大小的内部缓冲区来存储通过 IMFTransform::ProcessInput. This has the side effect that multiple samples must be provided as inputs before any output will be generated. Video encoders need access to multiple samples in order because they use the subsequent frames to determine how to encode the current frame. In other words, if the decoder is working on frame 0, it might need to see frames 1, 2, and 3 as well. From a technical standpoint, this is because of things like inter-frame prediction and motion estimation. Once the encoder is finished processing the oldest sample, it generates an output buffer (another IMFSample object, but this time on the output side via IMFTransform::ProcessOutput) then discards the input sample it was working on (by calling IUnknown::Release), then requests more input, and eventually moves on to the next frame. You can read more about this process in the MSDN article Processing Data in the Encoder
正如 Roman 所暗示的，这意味着您将 ID3D11Texture2D 封装在 IMFSample 中的 IMFMediaBuffer 中，然后将其传递给 Sink Writer。作为编码过程的一部分，该样本很可能被编码器缓冲。由于编码器正在工作，Texture2D 的内容可能会发生变化，这可能会导致各种问题。即使这不会导致程序错误，也肯定会导致非常奇怪的编码视频输出。想象一下，如果编码器试图预测一帧的视觉内容在下一帧中如何变化，然后两帧的实际视觉内容都从编码器下更新出来！
这个特定问题的发生是因为编码器只有一个指向您的 IMFSample 实例的指针引用，它最终只是指向您的 ID3D11Texture2D 对象的指针本身，并且该对象是一种指向可变图形内存的指针引用。最终，由于程序的某些其他部分，该图形内存的内容正在发生变化，但由于它总是更新相同的 GPU 纹理，因此您发送编码器的每个样本都指向相同的单个纹理。这意味着每当您更新纹理时，通过更改 GPU 内存，所有活动的 IMFSample 对象都将反映这些更改，因为它们都有效地指向相同的 GPU 纹理。
要解决此问题，您需要分配多个 ID3D11Texture2D 对象，以便在将其提供给 Sink Writer 时可以将一个纹理与一个 IMFSample 配对。这将通过使每个样本指向一个唯一的纹理来解决所有样本指向同一个 GPU 纹理的问题。但是，您不一定知道需要创建多少纹理，因此处理此问题的最安全方法是编写您自己的纹理分配器。这仍然可以在 C# 中完成，这是值得的，MediaFoundation.NET 定义了您需要使用的接口。
分配器应维护一个 "free" SharpDX.Texture2D 对象的列表 - 那些当前未被接收器写入器/编码器使用的对象。您的程序应该能够从分配器请求新的纹理对象，在这种情况下，它将 return 来自空闲列表的对象，或者创建一个新的纹理来满足请求。
下一个问题是知道 IMFSample 对象何时被编码器丢弃，以便您可以将附加的纹理添加回空闲列表。碰巧的是，您当前使用的 MFCreateVideoSampleFromSurface 函数分配实现 IMFTrackedSample 接口的样本。您将需要该接口，以便在释放示例时收到通知，以便您可以回收 Texture2D 个对象。
诀窍是你必须告诉样本你是分配者。首先，您的分配器 class 需要在编码器释放样本时实现作为参数传递的 IMFAsyncCallback. If you set your allocator class on the sample via IMFTrackedSample::SetAllocator, your allocator's IMFAsyncCallback::Invoke method will be called, with an IMFAsyncResult。这是分配器 class 的一般示例。
sealed class TextureAllocator : IMFAsyncCallback, IDisposable
{
private ConcurrentStack<SharpDX.Direct3D11.Texture2D> m_freeStack;
private static readonly Guid s_IID_ID3D11Texture2D = new Guid("6f15aaf2-d208-4e89-9ab4-489535d34f9c");
// If all textures are the exact same size and color format,
// consider making those parameters private class members and
// requiring they be specified as arguments to the constructor.
public TextureAllocator()
{
m_freeStack = new ConcurrentStack<SharpDX.Direct3D11.Texture2D>();
}
private bool disposedValue = false;
private void Dispose(bool disposing)
{
if(!disposedValue)
{
if(disposing)
{
// Dispose managed resources here
}
if(m_freeStack != null)
{
SharpDX.Direct3D11.Texture2D texture;
while(m_freeStack.TryPop(out texture))
{
texture.Dispose();
}
m_freeStack = null;
}
disposedValue = true;
}
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
~TextureAllocator()
{
Dispose(false);
}
private SharpDX.Direct3D11.Texture2D InternalAllocateNewTexture()
{
// Allocate a new texture with your format, size, etc here.
}
public SharpDX.Direct3D11.Texture2D AllocateTexture()
{
SharpDX.Direct3D11.Texture2D existingTexture;
if(m_freeStack.TryPop(out existingTexture))
{
return existingTexture;
}
else
{
return InternalAllocateNewTexture();
}
}
public IMFSample CreateSampleAndAllocateTexture()
{
IMFSample pSample;
IMFTrackedSample pTrackedSample;
HResult hr;
// Create the video sample. This function returns an IMFTrackedSample per MSDN
hr = MFExtern.MFCreateVideoSampleFromSurface(null, out pSample);
MFError.ThrowExceptionForHR(hr);
// Query the IMFSample to see if it implements IMFTrackedSample
pTrackedSample = pSample as IMFTrackedSample;
if(pTrackedSample == null)
{
// Throw an exception if we didn't get an IMFTrackedSample
// but this shouldn't happen in practice.
throw new InvalidCastException("MFCreateVideoSampleFromSurface returned a sample that did not implement IMFTrackedSample");
}
// Use our own class to allocate a texture
SharpDX.Direct3D11.Texture2D availableTexture = AllocateTexture();
// Convert the texture's native ID3D11Texture2D pointer into
// an IUnknown (represented as as System.Object)
object texNativeObject = Marshal.GetObjectForIUnknown(availableTexture.NativePointer);
// Create the media buffer from the texture
IMFMediaBuffer p2DBuffer;
hr = MFExtern.MFCreateDXGISurfaceBuffer(s_IID_ID3D11Texture2D, texNativeObject, 0, false, out p2DBuffer);
// Release the object-as-IUnknown we created above
COMBase.SafeRelease(texNativeObject);
// If media buffer creation failed, throw an exception
MFError.ThrowExceptionForHR(hr);
// Set the owning instance of this class as the allocator
// for IMFTrackedSample to notify when the sample is released
pTrackedSample.SetAllocator(this, null);
// Attach the created buffer to the sample
pTrackedSample.AddBuffer(p2DBuffer);
return pTrackedSample;
}
// This is public so any textures you allocate but don't make IMFSamples
// out of can be returned to the allocator manually.
public void ReturnFreeTexture(SharpDX.Direct3D11.Texture2D freeTexture)
{
m_freeStack.Push(freeTexture);
}
// IMFAsyncCallback.GetParameters
// This is allowed to return E_NOTIMPL as a way of specifying
// there are no special parameters.
public HResult GetParameters(out MFAsync pdwFlags, out MFAsyncCallbackQueue pdwQueue)
{
pdwFlags = MFAsync.None;
pdwQueue = MFAsyncCallbackQueue.Standard;
return HResult.E_NOTIMPL;
}
public HResult Invoke(IMFAsyncResult pResult)
{
object pUnkObject;
IMFSample pSample = null;
IMFMediaBuffer pBuffer = null;
IMFDXGIBuffer pDXGIBuffer = null;
// Get the IUnknown out of the IMFAsyncResult if there is one
HResult hr = pResult.GetObject(out pUnkObject);
if(Succeeded(hr))
{
pSample = pUnkObject as IMFSample;
}
if(pSample != null)
{
// Based on your implementation, there should only be one
// buffer attached to one sample, so we can always grab the
// first buffer. You could add some error checking here to make
// sure the sample has a buffer count that is 1.
hr = pSample.GetBufferByIndex(0, out pBuffer);
}
if(Succeeded(hr))
{
// Query the IMFMediaBuffer to see if it implements IMFDXGIBuffer
pDXGIBuffer = pBuffer as IMFDXGIBuffer;
}
if(pDXGIBuffer != null)
{
// Got an IMFDXGIBuffer, so we can extract the internal
// ID3D11Texture2D and make a new SharpDX.Texture2D wrapper.
hr = pDXGIBuffer.GetResource(s_IID_ID3D11Texture2D, out pUnkObject);
}
if(Succeeded(hr))
{
// If we got here, pUnkObject is the native D3D11 Texture2D as
// a System.Object, but it's unlikely you have an interface
// definition for ID3D11Texture2D handy, so we can't just cast
// the object to the proper interface.
// Happily, SharpDX supports wrapping System.Object within
// SharpDX.ComObject which makes things pretty easy.
SharpDX.ComObject comWrapper = new SharpDX.ComObject(pUnkObject);
// If this doesn't work, or you're using something like SlimDX
// which doesn't support object wrapping the same way, the below
// code is an alternative way.
/*
IntPtr pD3DTexture2D = Marshal.GetIUnknownForObject(pUnkObject);
// Create your wrapper object here, like this for SharpDX
SharpDX.ComObject comWrapper = new SharpDX.ComObject(pD3DTexture2D);
// or like this for SlimDX
SlimDX.Direct3D11.Texture2D.FromPointer(pD3DTexture2D);
Marshal.Release(pD3DTexture2D);
*/
// You might need to query comWrapper for a SharpDX.DXGI.Resource
// first, then query that for the SharpDX.Direct3D11.Texture2D.
SharpDX.Direct3D11.Texture2D texture = comWrapper.QueryInterface<SharpDX.Direct3D11.Texture2D>();
if(texture != null)
{
// Now you can add "texture" back to the allocator's free list
ReturnFreeTexture(texture);
}
}
}
}
在 Sink Writer 输入媒体类型上设置 MF_SA_D3D_AWARE
我不认为这会导致 HRESULT 您遇到的问题，但无论如何这都不是正确的做法。 MF_SA_D3D_AWARE（及其对应的 DX11，MF_SA_D3D11_AWARE）是由 IMFTransform 对象设置的属性，用于通知您转换分别通过 DX9 或 DX11 支持图形加速。无需在 Sink Writer 的输入媒体类型上进行设置。
在 texNativeObject
上没有 SafeRelease
我建议在 texNativeObject 上调用 COMBase.SafeRelease()，否则您可能会泄漏内存。那，或者你会不必要地延长那个 COM 对象的生命周期，直到 GC 为你清理引用计数
不必要的转换
这是您上面代码的一部分：
buffer = MFVideoEncoderST.ReinterpretCast<IMF2DBuffer,IMFMediaBuffer>(p2Dbuffer);
int length=0;
if (Succeeded(hr)) hr = (int)p2Dbuffer.GetContiguousLength(out length);
if (Succeeded(hr)) hr = (int)buffer.SetCurrentLength(length);
我不确定您的 ReinterpretCast 函数在做什么，但是如果您 确实 需要在 C# 中执行 QueryInterface 样式转换，您可以只使用 as 运算符或常规转换。
// pMediaBuffer is of type IMFMediaBuffer and has been created elsewhere
IMF2DBuffer p2DBuffer = pMediaBuffer as IMF2DBuffer;
if(p2DBuffer != null)
{
// pMediaBuffer is an IMFMediaBuffer that also implements IMF2DBuffer
}
else
{
// pMediaBuffer does not implement IMF2DBuffer
}
第一期：IMFDXGIDeviceManager::ResetDevice总是失败。
在我之前回答的评论中与@kripto 合作后，我们诊断出了许多其他问题。最大的问题是设置 IMFDXGIDeviceManager in order to enable a hardware H.264 encoder MFT to accept Direct3D11 Texture2D samples, contained inside an IMFDXGIBuffer。代码中有一个很难发现的错误：
// pDevice is a SharpDX.Direct3D11.Texture2D instance
// pDevice.NativePointer is an IntPtr that refers to the native IDirect3D11Device
// being wrapped by SharpDX.
IMFDXGIDeviceManager pDeviceManager;
object d3dDevice = Marshal.GetIUnknownForObject(pDevice.NativePointer);
HResult hr = MFExtern.MFCreateDXGIDeviceManager(out resetToken, out pDeviceManager);
if(Succeeded(hr))
{
// The signature of this is:
// HResult ResetDevice(object d3d11device, int resetToken);
hr = pDeviceManager.ResetDevice(d3dDevice, resetToken);
}
上面的代码是这样的。设备管理器已创建，但为了让编码器 MFT 访问 Texture2D 个样本，它需要创建纹理的同一 Direct3D 设备的副本。因此，您必须在设备管理器上调用 IMFDXGIDeviceManager::ResetDevice 才能为其提供 Direct3D 设备。有关 ResetDevice 的一些重要脚注，请参阅 [1]。 SharpDX 仅提供对指向本机 IDirect3D11Device 的 IntPtr 的访问，但 MediaFoundation.NET 接口需要传入 object 来代替。
看到错误了吗？上面的代码类型检查和编译都很好，但包含一个严重错误。错误是使用 Marshal.GetIUnknownForObject instead of Marshal.GetObjectForIUnknown。有趣的是，因为 object 可以很好地装箱 IntPtr，所以您可以使用完全相反的编组函数，它仍然可以很好地编译。问题是我们正在尝试将 IntPtr 转换为 object 内的 .NET RCW，这是 MediaFoundation.NET 中的 ResetDevice 所期望的。此错误导致 ResetDevice 到 return E_INVALIDARG 而不是正常工作。
第二期：奇怪的编码器输出
第二个问题是 Intel Quick Sync Video H.264 Encoder MFT 不是特别满意，虽然它被正确创建，但在结果文件的开头有一两秒的黑色输出，以及前几秒的阻塞和运动错误，有时视频的一半是灰色的，没有显示实际复制的桌面图像。
我想确保实际的 Texture2D 对象被正确发送到编码器，所以我写了一个简单的 class 将 Direct3D 11 Texture2D 转储到 .png 文件。我在这里为任何需要它的人提供了它——这需要 SharpDX 和 MediaFoundation.NET 才能工作，尽管您可以使用 CopyMemory in a loop to account for the different strides. Note that this is only set up to work with textures in DXGI.Format.B8G8R8A8_UNorm 格式删除 MF 依赖。它可能适用于其他格式的纹理，但输出看起来很奇怪。
using System;
using System.Drawing;
namespace ScreenCapture
{
class Texture2DDownload : IDisposable
{
private SharpDX.Direct3D11.Device m_pDevice;
private SharpDX.Direct3D11.Texture2D m_pDebugTexture;
public Texture2DDownload(SharpDX.Direct3D11.Device pDevice)
{
m_pDevice = pDevice;
}
/// <summary>
/// Compare all the relevant properties of the texture descriptions for both input textures.
/// </summary>
/// <param name="texSource">The source texture</param>
/// <param name="texDest">The destination texture that will have the source data copied into it</param>
/// <returns>true if the source texture can be copied to the destination, false if their descriptions are incompatible</returns>
public static bool TextureCanBeCopied(SharpDX.Direct3D11.Texture2D texSource, SharpDX.Direct3D11.Texture2D texDest)
{
if(texSource.Description.ArraySize != texDest.Description.ArraySize)
return false;
if(texSource.Description.Format != texDest.Description.Format)
return false;
if(texSource.Description.Height != texDest.Description.Height)
return false;
if(texSource.Description.MipLevels != texDest.Description.MipLevels)
return false;
if(texSource.Description.SampleDescription.Count != texDest.Description.SampleDescription.Count)
return false;
if(texSource.Description.SampleDescription.Quality != texDest.Description.SampleDescription.Quality)
return false;
if(texSource.Description.Width != texDest.Description.Width)
return false;
return true;
}
/// <summary>
/// Saves the contents of a <see cref="SharpDX.Direct3D11.Texture2D"/> to a file with name contained in <paramref name="filename"/> using the specified <see cref="System.Drawing.Imaging.ImageFormat"/>.
/// </summary>
/// <param name="texture">The <see cref="SharpDX.Direct3D11.Texture2D"/> containing the data to save.</param>
/// <param name="filename">The filename on disk where the output image should be saved.</param>
/// <param name="imageFormat">The format to use when saving the output file.</param>
public void SaveTextureToFile(SharpDX.Direct3D11.Texture2D texture, string filename, System.Drawing.Imaging.ImageFormat imageFormat)
{
// If the existing debug texture doesn't exist, or the incoming texture is different than the existing debug texture...
if(m_pDebugTexture == null || !TextureCanBeCopied(m_pDebugTexture, texture))
{
// Dispose of any existing texture
if(m_pDebugTexture != null)
{
m_pDebugTexture.Dispose();
}
// Copy the original texture's description...
SharpDX.Direct3D11.Texture2DDescription newDescription = texture.Description;
// Then modify the parameters to create a CPU-readable staging texture
newDescription.BindFlags = SharpDX.Direct3D11.BindFlags.None;
newDescription.CpuAccessFlags = SharpDX.Direct3D11.CpuAccessFlags.Read;
newDescription.OptionFlags = SharpDX.Direct3D11.ResourceOptionFlags.None;
newDescription.Usage = SharpDX.Direct3D11.ResourceUsage.Staging;
// Re-generate the debug texture by copying the new texture's description
m_pDebugTexture = new SharpDX.Direct3D11.Texture2D(m_pDevice, newDescription);
}
// Copy the texture to our debug texture
m_pDevice.ImmediateContext.CopyResource(texture, m_pDebugTexture);
// Map the debug texture's resource 0 for read mode
SharpDX.DataStream data;
SharpDX.DataBox dbox = m_pDevice.ImmediateContext.MapSubresource(m_pDebugTexture, 0, 0, SharpDX.Direct3D11.MapMode.Read, SharpDX.Direct3D11.MapFlags.None, out data);
// Create a bitmap that's the same size as the debug texture
Bitmap b = new Bitmap(m_pDebugTexture.Description.Width, m_pDebugTexture.Description.Height, System.Drawing.Imaging.PixelFormat.Format32bppRgb);
// Lock the bitmap data to get access to the native bitmap pointer
System.Drawing.Imaging.BitmapData bd = b.LockBits(new Rectangle(0, 0, b.Width, b.Height), System.Drawing.Imaging.ImageLockMode.WriteOnly, System.Drawing.Imaging.PixelFormat.Format32bppRgb);
// Use the native pointers to do a native-to-native memory copy from the mapped subresource to the bitmap data
// WARNING: This might totally blow up if you're using a different color format than B8G8R8A8_UNorm, I don't know how planar formats are structured as D3D textures!
//
// You can use Win32 CopyMemory to do the below copy if need be, but you have to do it in a loop to respect the Stride and RowPitch parameters in case the texture width
// isn't on an aligned byte boundary.
MediaFoundation.MFExtern.MFCopyImage(bd.Scan0, bd.Stride, dbox.DataPointer, dbox.RowPitch, bd.Width * 4, bd.Height);
/// Unlock the bitmap
b.UnlockBits(bd);
// Unmap the subresource mapping, ignore the SharpDX.DataStream because we don't need it.
m_pDevice.ImmediateContext.UnmapSubresource(m_pDebugTexture, 0);
data = null;
// Save the bitmap to the desired filename
b.Save(filename, imageFormat);
b.Dispose();
b = null;
}
#region IDisposable Support
private bool disposedValue = false; // To detect redundant calls
protected virtual void Dispose(bool disposing)
{
if(!disposedValue)
{
if(disposing)
{
}
if(m_pDebugTexture != null)
{
m_pDebugTexture.Dispose();
}
disposedValue = true;
}
}
// TODO: override a finalizer only if Dispose(bool disposing) above has code to free unmanaged resources.
~Texture2DDownload() {
// Do not change this code. Put cleanup code in Dispose(bool disposing) above.
Dispose(false);
}
// This code added to correctly implement the disposable pattern.
public void Dispose()
{
// Do not change this code. Put cleanup code in Dispose(bool disposing) above.
Dispose(true);
GC.SuppressFinalize(this);
}
#endregion
}
}
一旦我确认我有好的图像进入编码器，我发现代码没有调用 IMFSinkWriter::SendStreamTick after calling IMFSinkWriter::BeginWriting 而是在发送第一个 IMFSample 之前。初始样本也有一个非零的时间增量，这导致了初始黑色输出。为了解决这个问题，我添加了以下代码：
// Existing code to set the sample time and duration
// recordDuration is the current frame time in 100-nanosecond units
// prevRecordingDuration is the frame time of the last frame in
// 100-nanosecond units
sample.SetSampleTime(recordDuration);
sample.SetSampleDuration(recordDuration - prevRecordingDuration);
// The fix is here:
if(frames == 0)
{
sinkWriter.SendStreamTick(streamIndex, recordDuration);
sample.SetUINT32(MFAttributesClsid.MFSampleExtension_Discontinuity, 1);
}
sinkWriter.WriteSample(streamIndex, sample);
frames++;
通过向接收器编写器发送一个流标记，它确定 recordDuration 中的任何值现在都被视为输出视频流的时间 = 0 点。换句话说，一旦您调用 SetStreamTick 并传入帧时间戳，所有后续时间戳都会从中减去该初始时间戳。这就是让第一个样本帧立即显示在输出文件中的方法。
此外，无论何时调用SendStreamTick，之后直接提供给sink writer的样本必须在其属性上将MFSampleExtension_Discontinuity设置为1。这意味着发送的样本中存在间隙，并且传递给编码器的帧是该间隙之后的第一帧。这或多或少地告诉编码器从样本中制作一个关键帧，这可以防止我之前看到的运动和阻塞效果。
结果
实施这些修复后，我测试了应用程序并实现了 1920x1080 分辨率和每秒 60 帧输出的全屏捕获。比特率设置为 4096 kbit。 CPU 在英特尔 i7-4510U 笔记本电脑 CPU 上，大多数工作负载的使用率在 2.5% 到 7% 之间 - 极端的运动会使其达到约 10%。通过 SysInternals 的 Process Explorer 获得的 GPU 利用率在 1% 到 2% 之间。
[1] 我相信其中一些是 Direct3D 9 的遗留问题，当时 DirectX API 中没有很好地内置多线程支持，并且每当任何组件使用时都必须独占锁定设备它（即解码器、渲染器、编码器）。使用 D3D 9 你可以调用
ResetDevice 但之后就再也不能使用你自己的设备指针了。相反，您甚至必须在自己的代码中调用 LockDevice 和 UnlockDevice 来获取设备指针，因为 MFT 可能在同一时刻使用该设备。在 Direct3D 11 中，在 MFT 和控制应用程序中同时使用同一设备似乎没有问题 - 尽管如果发生任何随机崩溃，我建议阅读大量有关如何 IMFDXGIDeviceManager::LockDevice 和 UnlockDevice 工作并实施这些以确保设备始终受到完全控制。