如何在渲染回调中交错非交错的 AudioBufferList？

Question

我正在从事一个项目，该项目涉及使用 MTAudioProcessingTap 将音频从 AVPlayer 视频播放器对象流式传输到 libpd。对于tap的process loop，我参考了PdAudioUnits render callback代码；但我最近意识到 libpd 期望的音频格式与来自 tap 的音频不同——也就是说，tap 在传入的 AudioBufferList 中提供两个非交错音频数据缓冲区，而 libpd 期望交错样本。我不认为我可以改变水龙头本身来提供交错样本。

有谁知道我可以解决这个问题的方法吗？

我认为我需要以某种方式创建一个新的 AudioBufferList 或浮动缓冲区并将样本交错放置；但我不太确定该怎么做，而且看起来会很昂贵。如果有人能给我一些指点，我将不胜感激！

这是我安装水龙头的代码：

- (void)installTapWithItem:(AVPlayerItem *)playerItem {
    
    MTAudioProcessingTapCallbacks callbacks;
    
    callbacks.version = kMTAudioProcessingTapCallbacksVersion_0;
    callbacks.clientInfo = (__bridge void *)self;
    callbacks.init = tap_InitCallback;
    callbacks.finalize = tap_FinalizeCallback;
    callbacks.prepare = tap_PrepareCallback;
    callbacks.unprepare = tap_UnprepareCallback;
    callbacks.process = tap_ProcessCallback;
    
    MTAudioProcessingTapRef audioProcessingTap;
    if (noErr == MTAudioProcessingTapCreate(kCFAllocatorDefault, &callbacks, kMTAudioProcessingTapCreationFlag_PreEffects, &audioProcessingTap))
    {
        NSLog(@"Tap created!");
        
        AVAssetTrack *audioTrack = [playerItem.asset tracksWithMediaType:AVMediaTypeAudio].firstObject;
        AVMutableAudioMixInputParameters* inputParams = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:audioTrack];
        inputParams.audioTapProcessor = audioProcessingTap;
        
        AVMutableAudioMix* audioMix = [AVMutableAudioMix audioMix];
        audioMix.inputParameters = @[inputParams];
        playerItem.audioMix = audioMix;
    }
}

还有我的tap_ProcessCallback：

static void tap_ProcessCallback(MTAudioProcessingTapRef tap, CMItemCount numberFrames, MTAudioProcessingTapFlags flags, AudioBufferList *bufferListInOut, CMItemCount *numberFramesOut, MTAudioProcessingTapFlags *flagsOut)
{
    OSStatus status = MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, nil, numberFramesOut);
    if (noErr != status) {
        NSLog(@"Error: MTAudioProcessingTapGetSourceAudio: %d", (int)status);
        return;
    }
    
    TapProcessorContext *context = (TapProcessorContext *)MTAudioProcessingTapGetStorage(tap);
    
    // first, create the input and output ring buffers if they haven't been created yet
    if (context->frameSize != numberFrames) {
        NSLog(@"creating ring buffers with size: %ld", (long)numberFrames);
        createRingBuffers((UInt32)numberFrames, context);
    }
    
    //adapted from PdAudioUnit.m
    float *buffer = (float *)bufferListInOut->mBuffers->mData;
    
    if (context->inputRingBuffer || context->outputRingBuffer) {
        
        // output buffer info from ioData
        UInt32 outputBufferSize = bufferListInOut->mBuffers[0].mDataByteSize;
        UInt32 outputFrames = (UInt32)numberFrames;
        //        UInt32 outputChannels = bufferListInOut->mBuffers[0].mNumberChannels;
        
        // input buffer info from ioData *after* rendering input samples
        UInt32 inputBufferSize = outputBufferSize;
        UInt32 inputFrames = (UInt32)numberFrames;
        //        UInt32 inputChannels = 0;
        
        UInt32 framesAvailable = (UInt32)rb_available_to_read(context->inputRingBuffer) / context->inputFrameSize;
        while (inputFrames + framesAvailable < outputFrames) {
            // pad input buffer to make sure we have enough blocks to fill auBuffer,
            // this should hopefully only happen when the audio unit is started
            rb_write_value_to_buffer(context->inputRingBuffer, 0, context->inputBlockSize);
            framesAvailable += context->blockFrames;
        }
        rb_write_to_buffer(context->inputRingBuffer, 1, buffer, inputBufferSize);
        
        // input ring buffer -> context -> output ring buffer
        char *copy = (char *)buffer;
        while (rb_available_to_read(context->outputRingBuffer) < outputBufferSize) {
            rb_read_from_buffer(context->inputRingBuffer, copy, context->inputBlockSize);
            [PdBase processFloatWithInputBuffer:(float *)copy outputBuffer:(float *)copy ticks:1];
            rb_write_to_buffer(context->outputRingBuffer, 1, copy, context->outputBlockSize);
        }
        
        // output ring buffer -> audio unit
        rb_read_from_buffer(context->outputRingBuffer, (char *)buffer, outputBufferSize);
    }
}

Answer 1

回答我自己的问题...

我不确定为什么会这样，但确实如此。显然我也不需要使用环形缓冲区，这很奇怪。当 mNumberBuffers 只有一个缓冲区时，我还添加了一个开关。

if (context->frameSize && outputBufferSize > 0) {
    if (bufferListInOut->mNumberBuffers > 1) {
        float *left = (float *)bufferListInOut->mBuffers[0].mData;
        float *right = (float *)bufferListInOut->mBuffers[1].mData;
            
        //manually interleave channels
        for (int i = 0; i < outputBufferSize; i += 2) {
            context->interleaved[i] = left[i / 2];
            context->interleaved[i + 1] = right[i / 2];
        }
        [PdBase processFloatWithInputBuffer:context->interleaved outputBuffer:context->interleaved ticks:64];
        //de-interleave
        for (int i = 0; i < outputBufferSize; i += 2) {
            left[i / 2] = context->interleaved[i];
            right[i / 2] = context->interleaved[i + 1];
        }
    } else {
        context->interleaved = (float *)bufferListInOut->mBuffers[0].mData;
        [PdBase processFloatWithInputBuffer:context->interleaved outputBuffer:context->interleaved ticks:32];
    }
}

如何在渲染回调中交错非交错的 AudioBufferList？

How to interleave a non-interleaved AudioBufferList inside a render callback?

ios

objective-c

avfoundation

avplayer

libpd