Android MediaCodec，解码 MJPEG 帧
Android MediaCodec, decoding MJPEG frames
编辑：
没关系，我解决了我的问题。
//---------------------------------------- ----------------------------------
我在尝试使用 Android MediaCodec API 进行 MJPEG 流解码时遇到了一些问题。我想做的是从我的网络摄像头流式传输 jpeg 帧，并在我的 android 设备上使用硬件加速解码显示它们（目前我正在使用 Nvidia Shield 平板电脑，Android 6.0）。
现在我正在接收一个 jpeg 帧，我可以使用 BitmapFactory.decodeStream() 函数在屏幕上显示它。然而，我想做的是在专用硬件上解码图像，这样我可以获得更好的性能和更长的电池寿命。为此，我想将 MediaCodec API 与 'video/mjpeg' 解码器一起使用，我很确定我的设备上可以使用它（我检查了 MediaCodecList 和它就在 'OMX.Nvidia.mjpeg.decoder').
这个名字下面
我将我的 MediaCodec 与 SurfaceView 一起使用，几乎与在 'Grafika' 存储库中完成的方式相同。 (https://github.com/google/grafika)
这是解码器初始化代码，在surfaceCreated()回调中调用，无异常完成：
private Surface testSurface;
private MediaCodec decoder;
//....
private void initMediaApi() throws IOException {
decoder = MediaCodec.createDecoderByType("video/mjpeg");
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/mjpeg", 720, 480);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 25);
decoder.configure(mediaFormat, testSurface, null, 0);
decoder.start();
}
之后我有一个函数，每次收到新的 jpeg 帧时都会调用它。它有两个参数：一个带有 jpeg 图像的字节数组和该图像的字节大小：
private void testMediaApi(final byte[] frameBuffer, final int length) throws IOException {
final int TIMEOUT_USEC = 10000;
long firstInputTimeNsec = System.nanoTime();
int inputBufIndex = decoder.dequeueInputBuffer(TIMEOUT_USEC);
if (inputBufIndex < 0) {
return;
}
ByteBuffer inputBuffer = decoder.getInputBuffer(inputBufIndex);
inputBuffer.clear();
inputBuffer.put(frameBuffer, 0, length);
decoder.queueInputBuffer(inputBufIndex, 0, length, 0, 0);
boolean outputDone = false;
while (!outputDone) {
outputDone = true;
MediaCodec.BufferInfo mBufferInfo = new MediaCodec.BufferInfo();
int decoderStatus = decoder.dequeueOutputBuffer(mBufferInfo, 1000000);
Log.d(LOG_TAG, "decoderStatus: " + decoderStatus);
if (decoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) {
// no output available yet
Log.d(TAG, "no output from decoder available");
} else if (decoderStatus == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
// not important for us, since we're using Surface
Log.d(TAG, "decoder output buffers changed");
} else if (decoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
MediaFormat newFormat = decoder.getOutputFormat();
Log.d(TAG, "decoder output format changed: " + newFormat);
} else if (decoderStatus < 0) {
throw new RuntimeException("unexpected result from decoder.dequeueOutputBuffer: " + decoderStatus);
} else { // decoderStatus >= 0
if (firstInputTimeNsec != 0) {
// Log the delay from the first buffer of input to the first buffer
// of output.
long nowNsec = System.nanoTime();
Log.d(TAG, "startup lag " + ((nowNsec - firstInputTimeNsec) / 1000000.0) + " ms");
firstInputTimeNsec = 0;
}
Log.d(TAG, "surface decoder given buffer " + decoderStatus + " (size=" + mBufferInfo.size + ")");
if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
Log.d(TAG, "output EOS");
outputDone = true;
}
boolean doRender = (mBufferInfo.size != 0);
// As soon as we call releaseOutputBuffer, the buffer will be forwarded
// to SurfaceTexture to convert to a texture. We can't control when it
// appears on-screen, but we can manage the pace at which we release
// the buffers.
decoder.releaseOutputBuffer(decoderStatus, doRender);
Log.d(TAG, "Reached EOS, looping");
decoder.flush(); // reset decoder state
}
}
}
如您所见，我等待 1 秒完成一帧解码，这应该足够了。不幸的是，在第一次调用后我收到解码器状态代码 -1，这意味着输出数据仍然不可用。第二帧排队后我得到状态 -3 'decoder output buffers changed' 然后整个事情崩溃给我这个异常：
07-20 21:21:25.512 10096-10109/piotrek.androidfpvtest W/MediaCodec: ResourceManagerService died.
07-20 21:21:25.513 10096-10127/piotrek.androidfpvtest E/ACodec: OMX/mediaserver died, signalling error!
07-20 21:21:25.513 10096-10127/piotrek.androidfpvtest E/ACodec: signalError(omxError 0x8000100d, internalError -32)
07-20 21:21:25.513 10096-10126/piotrek.androidfpvtest E/MediaCodec: Codec reported err 0xffffffe0, actionCode 0, while in state 6
07-20 21:21:25.513 10096-10125/piotrek.androidfpvtest W/System.err: java.lang.IllegalStateException
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at android.media.MediaCodec.native_dequeueOutputBuffer(Native Method)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at android.media.MediaCodec.dequeueOutputBuffer(MediaCodec.java:2622)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at piotrek.androidfpvtest.MainActivity.testMediaApi(MainActivity.java:226)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at piotrek.androidfpvtest.MainActivity.doReceive(MainActivity.java:161)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at piotrek.androidfpvtest.MainActivity.access[=13=]0(MainActivity.java:29)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at piotrek.androidfpvtest.MainActivity.run(MainActivity.java:54)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at java.lang.Thread.run(Thread.java:818)
如果我等待第一个帧被解码而不是连续排队帧，程序将进入无限循环，输出如下所示：
07-20 21:58:29.970 10577-10641/piotrek.androidfpvtest I/OMXClient: Using client-side OMX mux.
07-20 21:58:30.110 10577-10640/piotrek.androidfpvtest I/MediaCodec: [OMX.Nvidia.mjpeg.decoder] setting surface generation to 10830849
07-20 21:58:30.115 10577-10641/piotrek.androidfpvtest I/ACodec: Enable timestamp filtering for Video Decoder
07-20 21:58:30.126 10577-10641/piotrek.androidfpvtest D/SurfaceUtils: set up nativeWindow 0xae47c708 for 720x480, color 0x106, rotation 0, usage 0x2b00
07-20 21:58:56.719 10577-10641/piotrek.androidfpvtest D/SurfaceUtils: set up nativeWindow 0xae47c708 for 720x480, color 0x147, rotation 0, usage 0x2b00
07-20 21:58:56.720 10577-10641/piotrek.androidfpvtest W/ACodec: [OMX.Nvidia.mjpeg.decoder] setting nBufferCountActual to 11 failed: -1010
07-20 21:58:56.720 10577-10641/piotrek.androidfpvtest W/ACodec: [OMX.Nvidia.mjpeg.decoder] setting nBufferCountActual to 10 failed: -1010
07-20 21:58:57.713 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:58:57.713 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:58:57.714 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -3
07-20 21:58:57.714 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoder output buffers changed
07-20 21:58:58.714 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:58:58.715 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:58:59.717 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:58:59.718 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:00.720 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:00.720 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:01.721 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:01.721 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:02.723 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:02.723 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:03.724 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:03.724 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:04.724 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:04.724 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:05.726 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:05.726 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:06.727 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:06.727 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
如果有人对此有任何经验API，我将非常感谢您的帮助！
我自己解决了我的问题。原来这个特殊的解码器不支持输出到surface，所以解决方案是获取raw ByteBuffer然后使用GLES做YUV->RGB转换并显示在屏幕上。
MediaCodec.createDecoderByType("video/mjpeg");会抛出 IllegalArgumentException Failed to initialize video/mjpeg, error 0xfffffffe
编辑： 没关系，我解决了我的问题。
//---------------------------------------- ----------------------------------
我在尝试使用 Android MediaCodec API 进行 MJPEG 流解码时遇到了一些问题。我想做的是从我的网络摄像头流式传输 jpeg 帧，并在我的 android 设备上使用硬件加速解码显示它们（目前我正在使用 Nvidia Shield 平板电脑，Android 6.0）。
现在我正在接收一个 jpeg 帧，我可以使用 BitmapFactory.decodeStream() 函数在屏幕上显示它。然而，我想做的是在专用硬件上解码图像，这样我可以获得更好的性能和更长的电池寿命。为此，我想将 MediaCodec API 与 'video/mjpeg' 解码器一起使用，我很确定我的设备上可以使用它（我检查了 MediaCodecList 和它就在 'OMX.Nvidia.mjpeg.decoder').
这个名字下面我将我的 MediaCodec 与 SurfaceView 一起使用，几乎与在 'Grafika' 存储库中完成的方式相同。 (https://github.com/google/grafika)
这是解码器初始化代码，在surfaceCreated()回调中调用，无异常完成：
private Surface testSurface;
private MediaCodec decoder;
//....
private void initMediaApi() throws IOException {
decoder = MediaCodec.createDecoderByType("video/mjpeg");
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/mjpeg", 720, 480);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 25);
decoder.configure(mediaFormat, testSurface, null, 0);
decoder.start();
}
之后我有一个函数，每次收到新的 jpeg 帧时都会调用它。它有两个参数：一个带有 jpeg 图像的字节数组和该图像的字节大小：
private void testMediaApi(final byte[] frameBuffer, final int length) throws IOException {
final int TIMEOUT_USEC = 10000;
long firstInputTimeNsec = System.nanoTime();
int inputBufIndex = decoder.dequeueInputBuffer(TIMEOUT_USEC);
if (inputBufIndex < 0) {
return;
}
ByteBuffer inputBuffer = decoder.getInputBuffer(inputBufIndex);
inputBuffer.clear();
inputBuffer.put(frameBuffer, 0, length);
decoder.queueInputBuffer(inputBufIndex, 0, length, 0, 0);
boolean outputDone = false;
while (!outputDone) {
outputDone = true;
MediaCodec.BufferInfo mBufferInfo = new MediaCodec.BufferInfo();
int decoderStatus = decoder.dequeueOutputBuffer(mBufferInfo, 1000000);
Log.d(LOG_TAG, "decoderStatus: " + decoderStatus);
if (decoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) {
// no output available yet
Log.d(TAG, "no output from decoder available");
} else if (decoderStatus == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
// not important for us, since we're using Surface
Log.d(TAG, "decoder output buffers changed");
} else if (decoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
MediaFormat newFormat = decoder.getOutputFormat();
Log.d(TAG, "decoder output format changed: " + newFormat);
} else if (decoderStatus < 0) {
throw new RuntimeException("unexpected result from decoder.dequeueOutputBuffer: " + decoderStatus);
} else { // decoderStatus >= 0
if (firstInputTimeNsec != 0) {
// Log the delay from the first buffer of input to the first buffer
// of output.
long nowNsec = System.nanoTime();
Log.d(TAG, "startup lag " + ((nowNsec - firstInputTimeNsec) / 1000000.0) + " ms");
firstInputTimeNsec = 0;
}
Log.d(TAG, "surface decoder given buffer " + decoderStatus + " (size=" + mBufferInfo.size + ")");
if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
Log.d(TAG, "output EOS");
outputDone = true;
}
boolean doRender = (mBufferInfo.size != 0);
// As soon as we call releaseOutputBuffer, the buffer will be forwarded
// to SurfaceTexture to convert to a texture. We can't control when it
// appears on-screen, but we can manage the pace at which we release
// the buffers.
decoder.releaseOutputBuffer(decoderStatus, doRender);
Log.d(TAG, "Reached EOS, looping");
decoder.flush(); // reset decoder state
}
}
}
如您所见，我等待 1 秒完成一帧解码，这应该足够了。不幸的是，在第一次调用后我收到解码器状态代码 -1，这意味着输出数据仍然不可用。第二帧排队后我得到状态 -3 'decoder output buffers changed' 然后整个事情崩溃给我这个异常：
07-20 21:21:25.512 10096-10109/piotrek.androidfpvtest W/MediaCodec: ResourceManagerService died.
07-20 21:21:25.513 10096-10127/piotrek.androidfpvtest E/ACodec: OMX/mediaserver died, signalling error!
07-20 21:21:25.513 10096-10127/piotrek.androidfpvtest E/ACodec: signalError(omxError 0x8000100d, internalError -32)
07-20 21:21:25.513 10096-10126/piotrek.androidfpvtest E/MediaCodec: Codec reported err 0xffffffe0, actionCode 0, while in state 6
07-20 21:21:25.513 10096-10125/piotrek.androidfpvtest W/System.err: java.lang.IllegalStateException
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at android.media.MediaCodec.native_dequeueOutputBuffer(Native Method)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at android.media.MediaCodec.dequeueOutputBuffer(MediaCodec.java:2622)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at piotrek.androidfpvtest.MainActivity.testMediaApi(MainActivity.java:226)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at piotrek.androidfpvtest.MainActivity.doReceive(MainActivity.java:161)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at piotrek.androidfpvtest.MainActivity.access[=13=]0(MainActivity.java:29)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at piotrek.androidfpvtest.MainActivity.run(MainActivity.java:54)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at java.lang.Thread.run(Thread.java:818)
如果我等待第一个帧被解码而不是连续排队帧，程序将进入无限循环，输出如下所示：
07-20 21:58:29.970 10577-10641/piotrek.androidfpvtest I/OMXClient: Using client-side OMX mux.
07-20 21:58:30.110 10577-10640/piotrek.androidfpvtest I/MediaCodec: [OMX.Nvidia.mjpeg.decoder] setting surface generation to 10830849
07-20 21:58:30.115 10577-10641/piotrek.androidfpvtest I/ACodec: Enable timestamp filtering for Video Decoder
07-20 21:58:30.126 10577-10641/piotrek.androidfpvtest D/SurfaceUtils: set up nativeWindow 0xae47c708 for 720x480, color 0x106, rotation 0, usage 0x2b00
07-20 21:58:56.719 10577-10641/piotrek.androidfpvtest D/SurfaceUtils: set up nativeWindow 0xae47c708 for 720x480, color 0x147, rotation 0, usage 0x2b00
07-20 21:58:56.720 10577-10641/piotrek.androidfpvtest W/ACodec: [OMX.Nvidia.mjpeg.decoder] setting nBufferCountActual to 11 failed: -1010
07-20 21:58:56.720 10577-10641/piotrek.androidfpvtest W/ACodec: [OMX.Nvidia.mjpeg.decoder] setting nBufferCountActual to 10 failed: -1010
07-20 21:58:57.713 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:58:57.713 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:58:57.714 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -3
07-20 21:58:57.714 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoder output buffers changed
07-20 21:58:58.714 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:58:58.715 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:58:59.717 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:58:59.718 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:00.720 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:00.720 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:01.721 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:01.721 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:02.723 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:02.723 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:03.724 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:03.724 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:04.724 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:04.724 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:05.726 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:05.726 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:06.727 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:06.727 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
如果有人对此有任何经验API，我将非常感谢您的帮助！
我自己解决了我的问题。原来这个特殊的解码器不支持输出到surface，所以解决方案是获取raw ByteBuffer然后使用GLES做YUV->RGB转换并显示在屏幕上。
MediaCodec.createDecoderByType("video/mjpeg");会抛出 IllegalArgumentException Failed to initialize video/mjpeg, error 0xfffffffe