MiniMax Music 3 latent re-planner

Experimental training code for editing MiniMax Music 3 DAV latents from a reference track.

Status: the architecture worked for restoration and small-set memorization. It did not generalize as a useful style-transfer model. The limiting factor was the training pairs, not another missing objective. New work should start with pairs whose target structure is predictable from the source.

The complete checkpoints, renders, metrics, and historical log are in minimax-music3-replanner-experiment. The restoration branch became minimax-music3-latent-refiner-v0.10.

Model

• 137-143M parameter flow-matching DiT in the full configuration.
• DAV latent rate: 86.13 Hz (44100 / 512).
• Reference structure: all 13 MERT-v1-95M hidden layers, injected per DiT layer.
• Style: pooled laion/larger_clap_music audio embedding.
• Optional source streams: eight RVQ codebooks and degraded DAV latents.
• Tasks: transfer, identity, restoration.
• Objectives retained for reproduction: flow matching, DDPM, bridge transport, in-context editing, MaskDiT frame masking, and the v8-v10 DPO ablations.
• One-process CUDA and torchrun DDP use the same entry point.

Result

The corpus contained 8,192 duration-matched source/ACE-Step style-transfer pairs and 64 holdouts.

Measured pair alignment explained the plateau:

• Source/target DAV latent cosine: 0.135.
• Frame-wise source/target MERT correlation: 0.233.
• Shuffled-pair MERT correlation: 0.16.

The model could restore audio because degraded and clean tracks were strongly aligned. The style-transfer targets were only weakly predictable from their sources. More training and objective changes did not fix that.

Candidate data for another attempt:

• Stem swaps while preserving the rest of the mix.
• MIDI transcription followed by controlled re-rendering.
• Covers with verified phrase, beat, and section alignment.

Files

• train_latent_replanner.py: canonical model, samplers, dataset, online feature extraction, trainer, DDP, evaluation, and all experimental objectives.
• latent_replanner_pipeline.py: checkpoint loader and MERT/CLAP/DAV inference pipeline for the released experiments.
• minimax_music3_dav.py: standalone full DAV encoder/decoder used by inference.
• precompute_rvq_codes.py: sharded Open-RVQ code export for the optional code stream. --raw-codes emits the format expected by MiniMax Music 3 LM training in SimpleTuner.
• comfyui_node/: ComfyUI loader and task-space generation nodes.
• comfyui_workflow_example.json: v10 transfer workflow.
• tests/test_train_latent_replanner.py: CPU contracts for model shapes, sampling, and pair splitting.

The three original trainers were consolidated here. Use --holdout-count 0 --batch-size 1 for the old single-pair mode. A short pair-ID file and a matching batch size replace the old few-pair mode.

Install

Python 3.12 or 3.13 is recommended. CUDA is required for training and online extraction.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

The RVQ exporter uses the standalone adapter from minimax-music3-rvq-reference-audio. The trainer does not require a SimpleTuner configuration.

Data layout

pair_ids.txt contains one relative audio path per line. The same path must exist below both roots.

source/
  artist/song.flac
target/
  artist/song.flac
pair_ids.txt

Do not split IDs on commas or spaces. Both are legal filename characters.

Train

Single GPU, v6-style in-context baseline:

python train_latent_replanner.py \
  --source-dir source \
  --target-dir target \
  --pair-ids-file pair_ids.txt \
  --output-dir outputs/v6 \
  --objective incontext \
  --task-conditioning \
  --identity-rate 0.15 \
  --degrade-rate 0.15 \
  --dpo-weight 0 \
  --holdout-count 64 \
  --batch-size 4

Eight GPUs:

torchrun --standalone --nproc-per-node 8 train_latent_replanner.py \
  --source-dir source \
  --target-dir target \
  --pair-ids-file pair_ids.txt \
  --output-dir outputs/v6-ddp \
  --objective incontext \
  --task-conditioning \
  --identity-rate 0.15 \
  --degrade-rate 0.15 \
  --dpo-weight 0 \
  --holdout-count 64 \
  --batch-size 4

--batch-size is per process. DPO is disabled by default. The on-policy DPO option is retained only to reproduce v8/v9.

RVQ codes

The code stream did not improve the original transfer experiment, but the exporter remains useful for ablation and LM datasets.

python precompute_rvq_codes.py \
  --source-dir source \
  --pair-ids-file pair_ids.txt \
  --output-dir rvq-codes \
  --device cuda:0

Run one process per device with matching --num-shards and distinct --shard-index values. Add --raw-codes for SimpleTuner LM training. Without it, per-book vocabulary offsets are applied for the replanner code embedding.

ComfyUI

The node reproduces the released task-space sampler. It defaults to v10 and downloads the selected checkpoint, normalization statistics, MERT, CLAP, and the full DAV encoder/decoder.

git clone https://github.com/bghira/minimax-music3-latent-replanner
ComfyUI/venv/bin/python -m pip install -r minimax-music3-latent-replanner/requirements.txt
ln -s "$PWD/minimax-music3-latent-replanner/comfyui_node" \
  ComfyUI/custom_nodes/minimax_music3_latent_replanner

Restart ComfyUI and load comfyui_workflow_example.json.

• source_audio supplies frame-aligned MERT structure and DAV context.
• Optional style_audio supplies the pooled CLAP embedding. Without it, the source supplies both structure and style.
• transfer, identity, and restore select the trained task embedding when the checkpoint has one.
• The v10 RVQ stream stays on its trained null embedding because the released renderer did not use source RVQ codes.
• Windowed generation defaults to the 30-second training length with two seconds of latent crossfade.

The node makes the historical weights usable. It does not change the result: transfer generalization remained weak. Use the latent refiner for the successful restoration model.

License

Apache-2.0. MERT and MiniMax Music 3 retain their own licenses.

Related MiniMax Music 3 work

• SimpleTuner MiniMax Music 3 quickstart
• Open RVQ encoders
• RVQ reference-audio integration
• Fiona Crapple LM LoRA
• Latent refiner and v0.10 weights
• Replanner experiment log