使用 xuggler 解码 audio/video 时音频断断续续

Question

所以我正在为在 libGDX 中工作的现有视频解码器编写音频解码器。问题是，当音频代码没有线程时，音频和视频是断断续续的。音频会播放一段，然后视频会播放一段。

我的解决方案是进行一些多线程处理，让视频内容发挥作用（因为 libGDX 渲染线程不是线程安全的，乱用它们肯定会导致坏事）。那么自然的选择是使用线程的东西来做音频。

这解决了视频断断续续的问题，但不仅音频仍然断断续续，而且到处都是伪影。

这是我第一次涉足严肃的音频编程，所以请记住，我可能不知道某些事情 basic.The 执行器服务是 SingleThreadExecutor，因为音频需要解码和写出有序。

更新方法如下：

public boolean update(float dtSeconds) {
    if(playState != PlayState.PLAYING) return false;

    long dtMilliseconds = (long)(dtSeconds * 1000);
    playTimeMilliseconds += dtMilliseconds;

    sleepTimeoutMilliseconds = (long) Math.max(0, sleepTimeoutMilliseconds - dtMilliseconds);
    if(sleepTimeoutMilliseconds > 0) {
        // The playhead is still ahead of the current frame - do nothing
        return false;
    }


    while(true) {
        int packet_read_result = container.readNextPacket(packet);

        if(packet_read_result < 0) {
            // Got bad packet - we've reached end of the video stream
            stop();
            return true;
        }


        if(packet.getStreamIndex() == videoStreamId) 
        {
            // We have a valid packet from our stream

            // Allocate a new picture to get the data out of Xuggler
            IVideoPicture picture = IVideoPicture.make(
                videoCoder.getPixelType(),
                videoCoder.getWidth(),
                videoCoder.getHeight()
            );

            // Attempt to read the entire packet
            int offset = 0;
            while(offset < packet.getSize()) {
                // Decode the video, checking for any errors
                int bytesDecoded = videoCoder.decodeVideo(picture, packet, offset);
                if (bytesDecoded < 0) {
                    throw new RuntimeException("Got error decoding video");
                }
                offset += bytesDecoded;

                /* Some decoders will consume data in a packet, but will not
                 * be able to construct a full video picture yet. Therefore
                 * you should always check if you got a complete picture
                 * from the decoder
                 */
                if (picture.isComplete()) {
                    // We've read the entire packet
                    IVideoPicture newPic = picture;

                    // Timestamps are stored in microseconds - convert to milli
                    long absoluteFrameTimestampMilliseconds = picture.getTimeStamp() / 1000;
                    long relativeFrameTimestampMilliseconds = (absoluteFrameTimestampMilliseconds - firstTimestampMilliseconds);
                    long frameTimeDelta = relativeFrameTimestampMilliseconds - playTimeMilliseconds;

                    if(frameTimeDelta > 0) {
                        // The video is ahead of the playhead, don't read any more frames until it catches up
                        sleepTimeoutMilliseconds = frameTimeDelta + sleepTolleranceMilliseconds;
                        return false;
                    }

                    /* If the resampler is not null, that means we didn't get the video in
                     * BGR24 format and need to convert it into BGR24 format
                     */
                    if (resampler != null) {
                        // Resample the frame
                        newPic = IVideoPicture.make(
                            resampler.getOutputPixelFormat(),
                            picture.getWidth(), picture.getHeight()
                        );

                        if (resampler.resample(newPic, picture) < 0) {
                            throw new RuntimeException("Could not resample video");
                        }
                    }

                    if (newPic.getPixelType() != IPixelFormat.Type.BGR24) {
                        throw new RuntimeException("Could not decode video" + " as BGR 24 bit data");
                    }

                    // And finally, convert the BGR24 to an Java buffered image
                    BufferedImage javaImage = Utils.videoPictureToImage(newPic);

                    // Update the current texture
                    updateTexture(javaImage);

                    // Let the caller know the texture has changed
                    return true;
                }
            }
        }
        else if(packet.getStreamIndex() == this.audioStreamId)
        {
            IAudioSamples samples = IAudioSamples.make(1024, audioCoder.getChannels());
            Thread thread = new Thread(new DecodeSoundRunnable(samples));
            thread.setPriority(Thread.MAX_PRIORITY);
            this.decodeThreadPool.execute(thread);

        }

    }

这是音频线程：

private class DecodeSoundRunnable implements Runnable
    {
        IAudioSamples samples;
        int offset = 0;
        IStreamCoder coder;

        public DecodeSoundRunnable(IAudioSamples samples)
        {
            this.samples = samples.copyReference();
            this.coder = audioCoder.copyReference();
        }

        @Override
        public void run() {
            while(offset < packet.getSize())
            {
                 int bytesDecoded = this.coder.decodeAudio(samples, packet, offset);
                 if (bytesDecoded < 0)
                    break;//throw new RuntimeException("got error decoding audio in: " + videoPath);

                 offset += bytesDecoded;                  
            }
            playJavaSound(samples, 0);
            //writeOutThreadPool.execute(new WriteOutSoundRunnable(samples, 0));

        }
    }

Answer 1

通过创建一个专用线程只写出音频数据来解决这个问题。这是有效的，因为 mLine.write(byte[] bytes) 在写入数据时会阻塞。

private class WriteOutSoundBytes implements Runnable
    {
        byte[] rawByte;
        public WriteOutSoundBytes(byte[] rawBytes)
        {
            rawByte = rawBytes;
        }
        @Override
        public void run() 
        {
            mLine.write(rawByte, 0, rawByte.length);
        }


    }

使用 xuggler 解码 audio/video 时音频断断续续

Choppy audio when decoding audio/video with xuggler

java

audio

audiovideoplayback

libgdx

xuggler