NAC audio tools

▶ Listen to every model — plots and decoded audio for all 13 encoders, built by build_site.py.

Paper, code and weights for every model: awesome-codec-architectures.

Batched, plot-ready encode / decode for neural audio codecs (NACs) and audio VAEs, behind one interface. Takes audio at any sample rate, returns every internal representation as a plottable [B, D, T] feature, and decodes codes or latents back to audio aligned with the input.

import librosa
from utils import encoder_models as em

waves, sr = librosa.load("song.wav", sr=None)   # mono [T] at the file's own sample rate

enc = em.DACEncoder()
out = enc.encode(waves, sr=sr)       # waves: audio as [T], [B, T] or [B, C, T]
out["codes"].plot()                  # any representation: .tensor, .shape, .fps, .plot()
audio = enc.decode(out["codes"])     # [B, C, T] at the input's sample rate and length
audio = enc.decode(out["quantized_z"])  # or the decoder's continuous input

See example.py for a tour and batch_example.py for encoding a folder of audio files.

Install

Python 3.10, torch 2.7.1 / CUDA 12.6. requirements.txt is the full pip freeze.

conda create -n audio_codecs python=3.10 pip -y
conda activate audio_codecs
export PYTHONNOUSERSITE=1   # keep ~/.local packages out

python -m pip install "numpy==1.26.4" "protobuf==3.19.6" "stable-audio-tools==0.0.20" "descript-audio-codec==1.0.0"
python -m pip install pytorch_lightning snac music2latent diffusers "git+https://github.com/facebookresearch/dacvae"

• SAO and SAME-L need pytorch_lightning. EnCodec and εar-VAE need nothing extra.
• Weights download from Hugging Face on first use.

Models

Frame rate is frames per second of the returned features. Every constructor also takes device (default "cuda:0") and batch_size (default 8 for the codecs, 4 for the autoencoders, 1 for SAME-L).

Shapes and decoding

Shape is .tensor.shape: B clips, D dimensions, T frames at the frame rate (for SNAC, its finest rate).

min / max / mean / sd were measured on one 6-second stereo drum clip (1_rock_87_beat_4-4.wav, Groove MIDI Dataset). Nothing is normalized: every representation is the model's own output on the model's own scale, so these are a sanity check for your own features, not fixed bounds. Two things follow from them:

• Scales are not comparable across models. εar-VAE sits at sd 0.56 and DAC's quantized_z at 3.65, about 6× apart, so standardize per model before feeding several into one downstream model, distance or colour scale.
• codes are raw codebook indices. The full range is 0 to codebook size − 1 (1024 for DAC and EnCodec 24 kHz, 2048 for EnCodec 32 kHz, 4096 for SNAC); the min and max above are only what this clip used.

The one representation carrying a scaling factor is ACE-Step v1 DCAE's latent, which has upstream MusicDCAE's shift and scale applied, as MusicDCAE.encode returns it. Everything else is the bare encoder output, except that std is softplus(scale), a standard deviation in natural units rather than the raw parameter. Input audio is resampled and padded, never peak- or loudness-normalized.

Decodable says what enc.decode() does with that representation:

• ✓ The model's own decoder takes it as it is. ✓ default is the one used when you pass the whole encode() output.
• ✓ quantized first The model can't decode it directly. decode() first snaps it to the codebook, the same step that produces codes, then decodes that. The audio is the same as decoding codes, not a more detailed version.
• ✓ normalized first SAME-L's decoder takes latent only. decode() first runs pre_softnorm through the model's SoftNorm, which turns it into latent, so the audio is the same as decoding latent.
• ✗ Can't be decoded; decode() raises a ValueError.

API

Every encoder has the same two methods:

Each EncodedFeature has:

Conventions

• Input: a single stereo clip is [1, 2, T]; [2, T] is two mono clips. Audio is resampled to the model's rate and right-padded to whole frames.
• Channels: mono models mix stereo down; stereo models get mono input as dual-mono.
• Seeding: item i uses seed + i, for VAE samples and the stochastic decoders (SNAC, SAME-L, Music2Latent, DACVAE). Results don't depend on batch_size. To reproduce item 1 on its own, pass seed=1.
• Determinism: TF32 is off and cuDNN is deterministic (with TF32, DAC flipped RVQ codes on identical input). Codes are bit-identical across runs and batch sizes; floats agree to about 1e-5, except SAME-L in fp16.

What the representations are

Every model turns audio into frames, one every 1 / frame rate seconds. They differ in what a frame holds.

Tokens (DAC, EnCodec, SNAC). Each frame becomes a few integers, each an index into a learned codebook. They're chosen in stages: the first codebook approximates the frame and each next one encodes what's still missing. Neighbouring indices aren't similar sounds. Tokens are what audio language models (e.g. MusicGen) predict.

Latents (continuous autoencoders). Each frame becomes a vector of real numbers. Nothing is rounded, so latents keep more detail, and similar sounds get nearby vectors.

• VAEs (SAO, DACVAE, ACE-Step 1.5, εar-VAE) describe each frame as a small cloud of likely points:

  Name	What it is
  mu	The cloud's centre, the best estimate. Deterministic: use it for analysis and decoding
  std	The cloud's width per dimension (the model's uncertainty). Usually small
  sample	mu + std × noise, one seeded point from the cloud. Diffusion models train on these. Sounds almost the same as mu

• Other autoencoders (SAME-L, Music2Latent, ACE-Step v1 DCAE) output one plain latent per frame. Also exposed: pre_softnorm (SAME-L, before the final normalization) and features (Music2Latent, before the bottleneck).

Fewer frames per second or fewer dimensions means more compression and less detail.

Model details

Each heading links to the model's entry in awesome-codec-architectures.

DAC — Descript Audio Codec

em.DACEncoder()                                     # 44.1 kHz model (default)
em.DACEncoder(weights="path/to/weights_44khz.pth")  # a local 44.1 kHz checkpoint

• encoder_z [B, 1024, T] · latents [B, 72, T] (9 codebooks × 8) · codes [B, 9, T] · quantized_z [B, 1024, T]
• Decodes from codes or quantized_z. Default: codes.
• encoder_z and latents decode too, but are quantized first, so they give the same audio as codes.

EnCodec

em.EnCodecEncoder()                                     # 32 kHz music model, MusicGen's tokenizer (default)
em.EnCodecEncoder(model_id="facebook/encodec_24khz")    # 24 kHz general-audio model

• encoder_z [B, 128, T] · codes [B, Q, T] · quantized_z [B, 128, T]
• Encodes at the highest bandwidth: Q = 4 codebooks at 32 kHz, 32 at 24 kHz.
• Decodes from codes or quantized_z. Default: codes.
• encoder_z decodes too, but is quantized first, so it gives the same audio as codes.
• The 48 kHz stereo model encodes in normalized 1 s chunks and is not supported.

SNAC — Multi-Scale Neural Audio Codec

em.SNACEncoder()                                      # 44 kHz music model (default)
em.SNACEncoder(model_id="hubertsiuzdak/snac_32khz")   # 32 kHz music model

• encoder_z [B, 1024, T] · codes [B, 4, T] · quantized_z [B, 1024, T]
• codes stacks 4 levels, coarse to fine. Level i has 8, 4, 2, 1 times fewer frames and is repeated to the finest rate, so codes[:, i, ::stride] recovers it.
• Decodes from codes or quantized_z. Default: codes. The decoder adds noise, so it is seeded per item.
• encoder_z decodes too, but is quantized first, so it gives the same audio as codes.

Stable Audio Open VAE

em.SAOEncoder()

• mu · std · sample [B, 64, T]
• Decodes from mu or sample. Default: mu.
• Loads only the VAE, not the diffusion model (utils/sao_vae.py).
• Gated on Hugging Face: request access, then log in with hf auth login.

SAME-L — Semantically-Aligned Music Autoencoder

em.SAMLEncoder()                                  # fp16 on CUDA, one clip at a time (default)
em.SAMLEncoder(half=False, batch_size=4)          # fp32: batch-invariant, ~3.4 GB of weights

• pre_softnorm [B, 256, T] (before SoftNorm) · latent [B, 256, T]
• Decodes from latent. The decoder draws random numbers, so it is seeded per item.
• pre_softnorm decodes too, but goes through SoftNorm first, so it gives the same audio as latent.
• Encoding is deterministic: the train-time mask noise is off.
• fp16 results shift by about 1% with batch_size, hence the default of 1. Use half=False for larger batches.

Music2Latent

em.Music2LatentEncoder()                          # one-step decoding (default)
em.Music2LatentEncoder(denoising_steps=3)         # more decoding steps

• features [B, 8192, T] (before the bottleneck) · latent [B, 64, T]
• Decodes from latent. Decoding starts from noise, so it is seeded per item.
• Runs in fp32: the package's fp16 autocast makes results batch-dependent and "can cause nans". This sets music2latent.inference.mixed_precision = False for the whole process.
• Weights download into the package's site-packages folder, not the Hugging Face cache.

DACVAE

em.DACVAEEncoder()                                # facebook/dacvae-watermarked

• mu · std · sample [B, 128, T]
• Decodes from mu or sample. Default: mu.
• The decoder embeds an AudioSeal watermark with a random 16-bit message, so it is seeded per item.

ACE-Step 1.5 VAE

em.ACEStep15Encoder()                             # the vae/ subfolder of ACE-Step/Ace-Step1.5

• mu · std · sample [B, 64, T]
• Decodes from mu or sample. Default: mu.
• Loads only the VAE, with diffusers.AutoencoderOobleck.

ACE-Step v1 music DCAE

em.ACEStepDCAEEncoder()                           # music_dcae_f8c8 + music_vocoder of ACE-Step-v1-3.5B

• latent [B, 128, T] (the model's [B, 8, 16, T] latent, flattened)
• Decodes latent to a mel spectrogram, then to audio with ACE-Step's HiFi-GAN vocoder.
• Downloads only the DCAE and vocoder, not the 3.5B generator. The mel front end and vocoder are vendored in third_party/ace_step, because the acestep package pins versions (e.g. transformers==4.50.0) that clash with this environment.

εar-VAE

em.EARVAEEncoder()                                # 48 kHz v2 model (default)
em.EARVAEEncoder(variant="ear_vae_44k")           # 44.1 kHz model

• mu · std · sample [B, 64, T]
• Decodes from mu or sample. Default: mu.
• The model code is imported from a pinned revision of the Hugging Face repo.
• The v2 config declares a decoder transformer that the checkpoint doesn't contain, so it is built only when its weights are present.

Verification

Each wrapper was checked against the model's own reference pipeline on the same input (relative L2 difference):

Every model was also round-tripped with 44.1 kHz stereo input, a different recording in each channel. Output length and rate matched the input, and each output channel matched its own input channel.