MiniMax Music 3 RVQ reference audio

Experimental reference-audio conditioning for MiniMax Music 3 using the open RVQ encoders.

The adapter converts reference audio into MiniMax-style RVQ codes. During language-model rollout, every Nth semantic c0 prediction is restricted to the encoder's top-five candidates. MiniMax selects the semantic token and generates the seven acoustic codebooks. The tested default is every fifth frame. The ComfyUI node also has an experimental continuation mode that uses the predicted RVQ sequence as language-model history before sampling a new segment.

This is not a trained cover or audio-to-audio model. It can influence timing and musical content, but reference adherence and prompt-driven transformation are inconsistent. Interval 1 tends toward reconstruction. Larger intervals give the language model more freedom and less reference structure.

Repository contents

• minimax_music3_reference_adapter.py: standalone RVQ encoder loader, Diffusers rollout, and pipeline patch.
• comfyui_open_rvq/: ComfyUI nodes.
• comfyui_workflow_example.json: tested interval-5 workflow.
• examples/diffusers_reference.py: command-line Diffusers example.
• experiments/: rollout benchmark and semantic-confidence diagnostics used during development.
• tests/test_reference_adapter.py: CPU contracts for alignment, model shapes, and adapter options.

Encoder weights stay in the Hugging Face collection repository. Use the v4 169M autoregressive-depth encoder unless reproducing an ablation.

Diffusers

git clone https://github.com/bghira/minimax-music3-rvq-reference-audio
cd minimax-music3-rvq-reference-audio

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .
python -m pip install \
  git+https://github.com/huggingface/diffusers@dafe3733fcfdbf3c48915fe77be3aef65b5d6a2d \
  transformers accelerate

Run the complete example:

python examples/diffusers_reference.py reference.flac output.flac \
  --prompt "heavy metal, distorted guitars, live drums" \
  --lyrics-file lyrics.txt \
  --reference-interval 5

The Python API is:

import soundfile as sf
import torch
from diffusers import ModularPipeline
from minimax_music3_reference_adapter import (
    MiniMaxMusic3ReferenceAdapter,
    install_diffusers_reference_adapter,
)

install_diffusers_reference_adapter()
pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-Music3")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

adapter = MiniMaxMusic3ReferenceAdapter.from_pretrained()
audio, sample_rate = sf.read("reference.flac", always_2d=True)
frame_hiddens, predicted_codes = adapter.encode_reference(
    pipe,
    torch.from_numpy(audio.T).float(),
    sample_rate,
    prompt="heavy metal, distorted guitars, live drums",
    lyrics="[instrumental]",
    device="cuda",
    reference_interval=5,
)

result = pipe(
    frame_hiddens=frame_hiddens,
    generator=torch.Generator(device="cpu").manual_seed(42),
    num_inference_steps=30,
    output_type="pt",
)

install_diffusers_reference_adapter() adds a frame_hiddens bypass to the MiniMax Music 3 modular pipeline. It does not replace model components.

ComfyUI

Clone directly into custom_nodes:

cd ComfyUI/custom_nodes
git clone https://github.com/bghira/minimax-music3-rvq-reference-audio
cd minimax-music3-rvq-reference-audio
../../.venv/bin/pip install -r requirements.txt

../../.venv/bin/hf download SimpleTuner/open-rvq-encoder-minimax-music3 \
  --include "encoders/*" \
  --local-dir .

../../.venv/bin/hf download MiniMaxAI/MiniMax-Music3 dav.pth \
  --local-dir ../../models/vae

Install the official ComfyUI MiniMax Music 3 diffusion model, text encoder, and DAV decoder separately. Restart ComfyUI and load comfyui_workflow_example.json. Select v4 in MiniMax Music3 RVQ Reference Encoder Loader.

The original dav.pth is required for reference encoding. The ComfyUI DAV file is decoder-only.

MiniMax Music3 Reference Audio Encode has two modes:

• reference retains the original behavior: it creates a segment with periodic top-five semantic constraints and the same duration as the reference.
• continuation feeds all eight predicted RVQ books into the language model as an audio prefix, then returns a newly generated segment of up to continuation_seconds. The model can stop earlier. The returned conditioning and seconds output cover only the new continuation, so concatenate the decoded result with the source audio separately if a single track is needed. reference_interval is ignored in this mode.

Continuation is also experimental. The substitute encoder's codes are approximate, and the diffusion decoder does not receive the source waveform or its boundary latents, so a seamless join is not guaranteed.

Method limits

• The RVQ encoders approximate a private encoder; they are not MiniMax weights.
• Real-audio generalization was not established by the reverse-distillation evaluation.
• Encoder context is 128 frames, or 5.12 seconds, with no cross-window state.
• Only semantic c0 is constrained. Acoustic codebooks c1 through c7 remain generated by MiniMax.
• Periodic top-five constraints were an inference experiment, not a trained reference-conditioning channel.
• Continuation mode is the exception to the constraint path above: it teacher-forces all eight predicted books only for the prefix, then MiniMax generates all eight books for the new segment.

Related MiniMax Music 3 work

• SimpleTuner MiniMax Music 3 quickstart
• Open RVQ encoder collection
• Fiona Crapple LM LoRA
• Latent refiner and v0.10 weights
• Latent replanner and experiment log

License

Repository code is Apache-2.0. MiniMax Music 3, encoder weights, and generated outputs retain their own terms.