[AUv3] Sidechain input handling can corrupt memory and ignores negotiated bus layouts

[spkfb]

[AUv3] Sidechain input handling can corrupt memory and ignores negotiated bus layouts

Summary

The AUv3 wrapper does not currently handle multi-channel input buses safely or consistently.

A typical configuration with:

• a stereo main input;
• a stereo sidechain input;
• a stereo output;

can cause an out-of-bounds write while constructing the AudioBufferList passed to pullInputBlock.

The wrapper also accepts host-negotiated channel counts that are not propagated to the internal YUP audio-bus layout. This can make the wrapper request the wrong number of channels, calculate incorrect channel offsets, or route sidechain audio incorrectly.

Severity

High / blocking for production AUv3 sidechain support

The issue may result in:

• stack corruption;
• undefined behaviour;
• crashes in AUv3 hosts;
• invalid sidechain routing;
• incorrect behaviour when the host negotiates a mono sidechain;
• real-time allocations in the render callback.

Affected area

modules/yup_audio_plugin_client/auv3/yup_audio_plugin_client_AUv3.mm

Current behaviour

The render callback creates a local AudioBufferList and then writes one AudioBuffer descriptor per channel:

AudioBufferList localBuffer = {};
localBuffer.mNumberBuffers = static_cast<UInt32> (numCh);

for (int ch = 0; ch < numCh && ch < 16; ++ch)
{
    localBuffer.mBuffers[ch].mNumberChannels = 1;
    localBuffer.mBuffers[ch].mDataByteSize =
        static_cast<UInt32> (frameCount * sizeof(float));
    localBuffer.mBuffers[ch].mData = scratchBuffer.getWritePointer(channelOffset + ch);
}

AudioBufferList is a variable-length structure. A stack declaration of a plain AudioBufferList only provides storage for its built-in first AudioBuffer.

Writing localBuffer.mBuffers[1] for a stereo bus therefore writes beyond the valid object storage.

This affects both the stereo main input and the stereo sidechain input.

Steps to reproduce

1. Create a YUP audio processor with:
   • input bus 0: stereo main input;
   • input bus 1: stereo sidechain input;
   • output bus 0: stereo output.
2. Build the processor as an AUv3 plugin.
3. Load it in an AUv3 host that exposes sidechain routing.
4. Connect a stereo signal to the main input.
5. Connect a different stereo signal to the sidechain input.
6. Start playback.
7. Run the host or plugin with AddressSanitizer enabled.

Actual result

One or more of the following may occur:

• AddressSanitizer reports a stack-buffer overflow;
• the host crashes;
• sidechain audio is missing or corrupted;
• the sidechain channels are routed to incorrect internal channels;
• behaviour changes depending on compiler optimisation or host implementation.

Expected result

• Every input bus should receive a correctly sized AudioBufferList.
• Main-input and sidechain channels should be routed deterministically.
• The wrapper should use the channel count actually negotiated with the host.
• No allocation or buffer resizing should occur in the real-time render callback.
• The plugin should safely support valid host-provided output buffers whose mData pointers are initially null.

Additional issues found

1. Negotiated channel counts are accepted but not applied

The bus-format validation currently appears to accept any channel count between one and the bus's declared maximum:

return newNumChannels > 0
    && newNumChannels <= bus.getNumChannels();

However, the render path continues to use the immutable YUP bus channel count when:

• sizing the scratch buffer;
• calculating channel offsets;
• creating input AudioBufferList instances;
• pulling audio from the host.

Example:

• the sidechain is declared as stereo;
• the host negotiates it as mono;
• the wrapper accepts the mono format;
• the render callback still treats the sidechain as stereo.

Until dynamic bus layouts are implemented, the safest behaviour is to accept only the declared channel count:

return newNumChannels == bus.getNumChannels();

A complete implementation should instead maintain a negotiated layout and use it consistently throughout resource allocation and rendering.

2. Output buffers with null mData are not handled

AUv3 hosts may provide an output AudioBufferList whose mData pointers are null. In that case, the audio unit must provide backing storage and update the output buffer descriptors.

Writing directly to outputData->mBuffers[ch].mData without checking for null may crash in a conforming host.

3. Scratch-buffer resizing may occur in the render callback

Calling scratchBuffer.setSize() from the render block can allocate memory or resize storage on the real-time audio thread.

The scratch buffer should be fully allocated in allocateRenderResourcesAndReturnError() using:

• maximumFramesToRender;
• the complete negotiated input layout;
• the complete negotiated output layout.

The render callback should only clear and reuse existing storage.

4. AU bus indexes may diverge from YUP bus indexes

If the YUP bus collection contains non-audio buses, using the original YUP bus index to access an AU audio-bus array can select the wrong bus.

The wrapper should maintain explicit mappings such as:

struct AudioBusMapping
{
    int yupBusIndex;
    int auBusIndex;
    int channelOffset;
};

Separate mappings should be maintained for input and output audio buses.

5. Channel capabilities do not represent the complete layout

For a processor with a stereo main input and a stereo sidechain, the complete input topology contains four input channels across two buses.

Reporting only the maximum number of channels present on one bus does not accurately describe the plugin's full input/output capabilities.

The AU channel-capability declaration should be reviewed against the complete multi-bus layout.

6. Bus names are not propagated

YUP bus names should be assigned to the corresponding AUAudioUnitBus instances so hosts can display meaningful labels such as:

• Main Input;
• Sidechain;
• Output.

Suggested implementation

Allocate valid AudioBufferList storage

Preallocate one variable-sized AudioBufferList per audio bus outside the render callback.

Possible approaches include:

• AUAudioUnitBusArray-compatible render resources;
• AVAudioFormat and AVAudioPCMBuffer;
• AudioBufferList storage allocated with enough trailing AudioBuffer entries;
• a dedicated RAII wrapper around AudioBufferList.

The storage must have capacity for the negotiated number of buffers on the bus.

Store the negotiated layout

During resource allocation:

1. read the current format of every AU input and output bus;
2. validate the format against the processor's supported layouts;
3. store the negotiated channel count for each bus;
4. calculate stable channel offsets;
5. size all scratch storage once;
6. prepare the processor with the resulting layout.

Keep the render callback real-time safe

The render callback should not:

• allocate memory;
• resize containers;
• create variable-sized temporary structures;
• acquire locks;
• perform Objective-C operations that may allocate.

Handle null output pointers

When the host provides null output data pointers, use preallocated output storage and update the supplied AudioBufferList descriptors before processing or copying audio.

Proposed tests

Memory-safety tests

• Stereo main input with no sidechain.
• Stereo main input plus mono sidechain.
• Stereo main input plus stereo sidechain.
• Maximum supported channel count.
• AddressSanitizer enabled.
• UndefinedBehaviorSanitizer enabled.

Routing tests

Send distinct constant values to every input channel:

Bus	Channel	Test value
Main	Left	0.1
Main	Right	0.2
Sidechain	Left	0.3
Sidechain	Right	0.4

Verify that the processor receives the values in the expected internal channels and that no channels overlap.

Format-negotiation tests

• Reject unsupported mono/stereo changes when using an immutable layout.
• Accept and apply supported changes when dynamic layouts are implemented.
• Confirm that offsets are recalculated after a layout change.
• Confirm that resource reallocation occurs outside the render callback.

Host-buffer tests

• Host-provided non-null output buffers.
• Host-provided output buffers with null mData.
• Interleaved format rejection or conversion, depending on the supported contract.
• Sidechain disconnected.
• Sidechain connected after playback starts.
• Sidechain removed while the audio unit is stopped and resources are reallocated.

Acceptance criteria

• No out-of-bounds access when handling a bus with more than one channel.
• AddressSanitizer reports no error for stereo main and stereo sidechain rendering.
• The negotiated channel count is used consistently for every AU bus.
• Main and sidechain channels are routed to stable, documented internal offsets.
• No memory allocation or buffer resizing occurs in the render callback.
• Null host output pointers are handled correctly.
• Audio-only AU bus indexes are explicitly mapped to YUP bus indexes.
• Bus names are visible in compatible hosts.
• Automated tests cover mono and stereo sidechain configurations.
• The implementation passes validation in at least one AUv3 host with sidechain support.