从 GL_TEXTURE_EXTERNAL_OES 读取到 GL_TEXTURE_2D 有性能问题和故障

Question

我需要将数据从 GL_TEXTURE_EXTERNAL_OES 发送到简单的 GL_TEXTURE_2D（将图像从 Android 播放器渲染到 Unity 纹理），目前通过从缓冲区读取像素来完成源纹理。此过程在我的 OnePlus 5 phone 上正常运行，但在 phone 上的图像有一些故障，如小米 note 4、mi a2 等（例如图像非常绿），并且还有性能问题因为这个过程在每一帧都有效，而且要读取的像素更多，所以性能更差（即使我的 phone 在 4k 分辨率下 fps 也很低）。知道如何优化此过程或以其他方式进行吗？

谢谢并致以最诚挚的问候！

GLuint FramebufferName;
glGenFramebuffers(1, &FramebufferName);
glBindFramebuffer(GL_FRAMEBUFFER, FramebufferName);

glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_EXTERNAL_OES, g_ExtTexturePointer, 0);

if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE)
{
    LOGD("%s", "Error: Could not setup frame buffer.");
}

unsigned char* data = new unsigned char[g_SourceWidth * g_SourceHeight * 4];
glReadPixels(0, 0, g_SourceWidth, g_SourceHeight, GL_RGBA, GL_UNSIGNED_BYTE, data);

glBindTexture(GL_TEXTURE_2D, g_TexturePointer);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, g_SourceWidth, g_SourceHeight, 0, GL_RGBA, GL_UNSIGNED_BYTE, data);

glDeleteFramebuffers(1, &FramebufferName);
glBindFramebuffer(GL_FRAMEBUFFER, 0);
glBindTexture(GL_TEXTURE_2D, 0);

delete[] data;

更新。 包含此代码的函数和从 Unity 端调用它的函数

static void UNITY_INTERFACE_API OnRenderEvent(int eventID) { ... }

extern "C" UnityRenderingEvent UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API UMDGetRenderEventFunc()
{
    return OnRenderEvent;
}

从 Unity Update 函数调用的是这样的：

[DllImport("RenderingPlugin")]
static extern IntPtr UMDGetRenderEventFunc();

IEnumerator UpdateVideoTexture()
{
    while (true)
    {
        ...
        androidPlugin.UpdateSurfaceTexture();
        GL.IssuePluginEvent(UMDGetRenderEventFunc, 1);
    } 
}

并且 Android 插件在其一侧执行此操作（surfaceTexture 其纹理包含 ExoPlayer 在其上渲染视频的外部纹理）

public void exportUpdateSurfaceTexture() {
    synchronized (this) {
        if (this.mIsStopped) {
            return;
        }
        surfaceTexture.updateTexImage();
    }
}

Answer 1

在 C++ 方面：

当您执行 new unsigned char[g_SourceWidth * g_SourceHeight * 4]; 和 delete[] data 时，您每帧都在创建和销毁像素数据，这取决于纹理大小。创建纹理数据一次，然后重新使用它。

实现此目的的一种方法是在 C++ 端让 static 变量保存纹理信息，然后使用一个函数来初始化这些变量::

static void* pixelData = nullptr;
static int _x;
static int _y;
static int _width;
static int _height;

void initPixelData(void* buffer, int x, int y, int width, int height) {
    pixelData = buffer;
    _x = x;
    _y = y;
    _width = width;
    _height = height;
}

那么你的捕获函数应该被重写以删除 new unsigned char[g_SourceWidth * g_SourceHeight * 4]; 和 delete[] data 但使用静态变量。

static void UNITY_INTERFACE_API OnRenderEvent(int eventID)
{
    if (pixelData == nullptr) {
        //Debug::Log("Pointer is null", Color::Red);
        return;
    }

    GLuint FramebufferName;
    glGenFramebuffers(1, &FramebufferName);
    glBindFramebuffer(GL_FRAMEBUFFER, FramebufferName);

    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_EXTERNAL_OES, g_ExtTexturePointer, 0);

    if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE)
    {
        LOGD("%s", "Error: Could not setup frame buffer.");
    }

    glReadPixels(_x, _y, _width, _height, GL_RGBA, GL_UNSIGNED_BYTE, pixelData);

    glBindTexture(GL_TEXTURE_2D, g_TexturePointer);
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, _width, _height, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData);

    glDeleteFramebuffers(1, &FramebufferName);
    glBindFramebuffer(GL_FRAMEBUFFER, 0);
    glBindTexture(GL_TEXTURE_2D, 0);
}

extern "C" UnityRenderingEvent UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API
UMDGetRenderEventFunc()
{
    return OnRenderEvent;
}

在 C# 方面：

[DllImport("RenderingPlugin", CallingConvention = CallingConvention.Cdecl)]
public static extern void initPixelData(IntPtr buffer, int x, int y, int width, int height);

[DllImport("RenderingPlugin", CallingConvention = CallingConvention.StdCall)]
private static extern IntPtr UMDGetRenderEventFunc();

创建纹理信息，将其固定并将指针发送到 C++：

int width = 500;
int height = 500;

//Where Pixel data will be saved
byte[] screenData;
//Where handle that pins the Pixel data will stay
GCHandle pinHandler;

//Used to test the color
public RawImage rawImageColor;
private Texture2D texture;

// Use this for initialization
void Awake()
{
    Resolution res = Screen.currentResolution;
    width = res.width;
    height = res.height;

    //Allocate array to be used
    screenData = new byte[width * height * 4];
    texture = new Texture2D(width, height, TextureFormat.RGBA32, false, false);

    //Pin the Array so that it doesn't move around
    pinHandler = GCHandle.Alloc(screenData, GCHandleType.Pinned);

    //Register the screenshot and pass the array that will receive the pixels
    IntPtr arrayPtr = pinHandler.AddrOfPinnedObject();

    initPixelData(arrayPtr, 0, 0, width, height);

    StartCoroutine(UpdateVideoTexture());
}

然后要更新纹理，请参见下面的示例。请注意，有两种方法可以更新纹理，如下面的代码所示。如果您运行遇到 Method1 的问题，注释掉 使用 texture.LoadRawTextureData 和 texture.Apply 和 un-comment[ 的两行=48=] 使用 ByteArrayToColor、texture.SetPixels 和 texture.Apply 函数的 Method2 代码：

IEnumerator UpdateVideoTexture() { while (true) { //Take screenshot of the screen GL.IssuePluginEvent(UMDGetRenderEventFunc(), 1); //Update Texture Method1 texture.LoadRawTextureData(screenData); texture.Apply(); //Update Texture Method2. Use this if the Method1 above crashes /* ByteArrayToColor(); texture.SetPixels(colors); texture.Apply(); */ //Test it by assigning the texture to a raw image rawImageColor.texture = texture; //Wait for a frame yield return null; } } Color[] colors = null; void ByteArrayToColor() { if (colors == null) { colors = new Color[screenData.Length / 4]; } for (int i = 0; i < screenData.Length; i += 4) { colors[i / 4] = new Color(screenData[i], screenData[i + 1], screenData[i + 2], screenData[i + 3]); } }

完成后或脚本即将销毁时取消固定阵列：

void OnDisable() { //Unpin the array when disabled pinHandler.Free(); }

Answer 2

调用glReadPixels总是很慢； CPU 不擅长批量数据传输。

理想情况下，您设法说服 Unity 接受外部图像句柄，并执行整个过程零复制，但如果失败，我会使用 GPU 渲染到纹理并使用着色器从外部传输图像到 RGB 表面。

从 GL_TEXTURE_EXTERNAL_OES 读取到 GL_TEXTURE_2D 有性能问题和故障

Read from GL_TEXTURE_EXTERNAL_OES to GL_TEXTURE_2D have perfomance issues and glitches

android

opengl-es

framebuffer

unity3d

exoplayer