MediaCodec 不解码 H264 编码的视频数据

Question

我正在开发一个通过 RTP 接收 H264 编码数据的应用程序，但我无法让 Android 的 MediaCodec 输出任何内容。我正在按照此处所述

对 RTP 数据包进行拆包

编码帧重新组合后，我将它们送入出队输入缓冲区。

我对输入缓冲区进行排队时没有收到任何错误，但是解码器的回调从未调用方法 onOutputBufferAvailable。

我能够调用它的唯一方法是传递流结束标志，然后输出大小为 0。

我的问题是，“有什么我遗漏的明显错误吗？”和 'what potential issues could lead to output buffers never becoming available, but the codec not throwing an error?'

代码已更新

LinkedList<DatagramPacket> packets = new LinkedList<>();
Thread socketReader = new Thread(() -> {
        try {
            DatagramSocket socket  = new DatagramSocket(videoPort);
            socket.connect(InetAddress.getByName(remoteAddress),remotePort);
            byte[] b;
            while (true){
                b = new byte[1500];
                DatagramPacket p = new DatagramPacket(b,1500);
                socket.receive(p);
                //Log.d(TAG,"RTP: "+bytesToHex(p.getData()));
                packets.add(p);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    });
@SuppressLint({"NewApi", "LocalSuppress"}) Thread packetHandler = new Thread(() -> {
        try {
            MediaCodec decoder = MediaCodec.createDecoderByType(MIME_TYPE);
            MediaFormat decoderFormat = MediaFormat.createVideoFormat(MIME_TYPE, 1280, 720);
            LinkedList<ByteBuffer> decoderInputs = new LinkedList<>();
            LinkedList<Integer> decoderIndices = new LinkedList<>();

            decoder.setCallback(new MediaCodec.Callback() {
                @Override
                public void onInputBufferAvailable(@NonNull MediaCodec codec, int index) {
                    decoderInputs.add(codec.getInputBuffer(index));
                    decoderIndices.add(new Integer(index));
                }

                @Override
                public void onOutputBufferAvailable(@NonNull MediaCodec codec, int index, @NonNull MediaCodec.BufferInfo info) {
                    Log.d(TAG,"OUTPUT AVAILABLE!!!!");
                    Log.d(TAG, String.valueOf(info.size));
                }

                @Override
                public void onError(@NonNull MediaCodec codec, @NonNull MediaCodec.CodecException e) {
                    Log.e(TAG,e.toString());
                }
                @Override
                public void onOutputFormatChanged(@NonNull MediaCodec codec, @NonNull MediaFormat format) {
                    Log.d(TAG,"FORMAT CHANGED");
                }
            });
            decoder.configure(decoderFormat, null,null,0);
            decoder.start();
            RTPFrame frame = new RTPFrame();
            boolean sps = false;
            boolean pps = false;
            while (true){
                if(decoderIndices.peek()!=null){
                    DatagramPacket packet = packets.poll();
                    if(packet!=null){
                        byte[] data = packet.getData();
                        if(frame==null||(!frame.isInitialized())){
                            frame = new RTPFrame(data);
                        }
                        else{
                            if(frame.isComplete()){
                                Integer index = null;
                                ByteBuffer decoderInput = null;
                                try {
                                    decoderInput = decoderInputs.poll().put(frame.getFrame());
                                    index = decoderIndices.poll();
                                    int size = frame.getFrameSize();
                                    if (frame.SPS) {
                                        Log.d(TAG,"Depacketized at "+Integer.toUnsignedString(frame.getTime())+" length = "+frame.getFrameSize()+": " + bytesToHex(frame.getFrame().array()));
                                        decoder.queueInputBuffer(index, 0, size, frame.getTime(), MediaCodec.BUFFER_FLAG_CODEC_CONFIG);
                                        sps = true;
                                        pps = false;
                                    } else if (frame.PPS) {
                                        if(sps){
                                            Log.d(TAG,"Depacketized at "+Integer.toUnsignedString(frame.getTime())+" length = "+frame.getFrameSize()+": " + bytesToHex(frame.getFrame().array()));
                                            decoder.queueInputBuffer(index, 0, size, frame.getTime(), MediaCodec.BUFFER_FLAG_CODEC_CONFIG);
                                            pps = true;
                                        }
                                    } else if (!frame.badFrame) {
                                        if(sps&&pps){
                                            Log.d(TAG,"Depacketized at "+Integer.toUnsignedString(frame.getTime())+" length = "+frame.getFrameSize()+": " + bytesToHex(frame.getFrame().array()));
                                            decoder.queueInputBuffer(index, 0, size, frame.getTime(), 0);
                                        }
                                    }
                                    else{
                                        throw new RuntimeException();
                                    }
                                }
                                catch(Exception e){
                                    e.printStackTrace();
                                    if(index!=null){
                                        decoderIndices.push(index);
                                    }
                                    decoderInput.clear();
                                    decoderInputs.push(decoderInput);
                                }
                                frame = new RTPFrame();
                            }
                            else{
                                frame.addNALUnit(data);
                            }
                        }
                        if(frame.isComplete()){
                            Integer index = null;
                            ByteBuffer decoderInput = null;
                            try {
                                decoderInput = decoderInputs.poll().put(frame.getFrame());
                                index = decoderIndices.poll();
                                int size = frame.getFrameSize();
                                if (frame.SPS) {
                                    Log.d(TAG,"Depacketized at "+Integer.toUnsignedString(frame.getTime())+" length = "+frame.getFrameSize()+": " + bytesToHex(frame.getFrame().array()));
                                    decoder.queueInputBuffer(index, 0, size, frame.getTime(), MediaCodec.BUFFER_FLAG_CODEC_CONFIG);
                                    sps = true;
                                    pps = false;
                                } else if (frame.PPS) {
                                    if(sps){
                                        Log.d(TAG,"Depacketized at "+Integer.toUnsignedString(frame.getTime())+" length = "+frame.getFrameSize()+": " + bytesToHex(frame.getFrame().array()));
                                        decoder.queueInputBuffer(index, 0, size, frame.getTime(), MediaCodec.BUFFER_FLAG_CODEC_CONFIG);
                                        pps = true;
                                    }
                                } else if (!frame.badFrame) {
                                    if(sps&&pps){
                                        Log.d(TAG,"Depacketized at "+Integer.toUnsignedString(frame.getTime())+" length = "+frame.getFrameSize()+": " + bytesToHex(frame.getFrame().array()));
                                        decoder.queueInputBuffer(index, 0, size, frame.getTime(), 0);
                                    }
                                }
                                else{
                                    throw new Exception("bad frame");
                                }
                            }
                            catch(Exception e){
                                e.printStackTrace();
                                if(index!=null){
                                    decoderIndices.push(index);
                                }
                                decoderInput.clear();
                                decoderInputs.push(decoderInput);
                            }
                            frame = new RTPFrame();
                        }
                    }
                }
            }
        } catch (Exception e) {e.printStackTrace();}
    });

这是我收到的 1 帧所有 NAL 单元的示例，以及我如何对它们进行解包。

RTP: 80 63 00 2D 27 0E 64 30 66 B4 BA 42 3C 81 E0 00 80 6F...

RTP: 80 63 00 2E 27 0E 64 30 66 B4 BA 42 3C 01 F3 0F E0 3F...

RTP: 80 63 00 2F 27 0E 64 30 66 B4 BA 42 3C 01 37 9B FA BA...

RTP: 80 63 00 30 27 0E 64 30 66 B4 BA 42 3C 01 7F EA 75 7C...

RTP: 80 63 00 31 27 0E 64 30 66 B4 BA 42 3C 01 FA D8 A9 FF...

RTP: 80 63 00 32 27 0E 64 30 66 B4 BA 42 3C 01 1B C5 BC C0...

RTP: 80 63 00 33 27 0E 64 30 66 B4 BA 42 3C 01 0F F4 9A DE...

RTP: 80 63 00 34 27 0E 64 30 66 B4 BA 42 3C 01 F4 35 CD 28...

RTP: 80 63 00 35 27 0E 64 30 66 B4 BA 42 3C 01 9E 45 70 13...

RTP: 80 E3 00 36 27 0E 64 30 66 B4 BA 42 3C 41 0F 18 0D 83...

D/RTPreader: Depacketized at 2985639104 length = 12611: 00 00 00 01 21 E0 00 80 6F F0 B4 24 CD 5F 45 80 79 6E 0C...

这是解包前后的 SPS 和 PPS 示例

RTP: 80 63 00 00 5F DF A1 70 4F 2F 8A 3E 27 42 00 1F 8D 68 05 00 5B A1 00 00 03 00 01 00 00 03 00 1E 0F 10 7A 80

Depacketized at 1608491376 length = 27: 00 00 01 27 42 00 1F 8D 68 05 00 5B A1 00 00 03 00 01 00 00 03 00 1E 0F 10 7A 80

Depacketized at 1608491376 length = 7: 00 00 01 28 CE 32 48

RTP: 80 63 00 01 5F DF A1 70 4F 2F 8A 3E 28 CE 32 48

Answer 1

感谢@greeble31 和@ChrisBe 帮助解决问题。问题确实出在我的 RTPFrame class 上。具体来说，方法 getFrame()，它遍历包含 NAL 单元数据的 ByteBuffer 列表，并使用 put(ByteBuffer) 将它们添加到 bb。这增加了 bb 的 position，直到它等于 limit。

public ByteBuffer getFrame() {
    size = 4;
    int nalCount = nalUnits.size();
    for (int i = 0;i<nalCount;i++){
        size+=nalUnits.get(i).data.length;
    }

    ByteBuffer bb = ByteBuffer.allocate(size);
    bb.put(new byte[]{0,0,0,1});
    for(NALUnit unit:nalUnits){
        bb.put(unit.data);
    }
    return bb;
}

将 return bb; 更改为 return (ByteBuffer)bb.position(0); 解决了该问题。

MediaCodec 不解码 H264 编码的视频数据

MediaCodec not decoding H264 encoded video data

android

rtp

h.264

android-mediacodec