WWWWxp/awesome-continuous-audio-vae

A curated catalog of continuous audio VAEs and autoencoders for audio, music, singing, and speech.

0 commits

updated Sep 4, 2026

See the code

README

🎧 Awesome Continuous Audio VAE

A curated catalog of continuous audio VAEs and autoencoders for audio, music, singing, and speech.

Audio / Sound · Music · Singing · Speech

Papers, official implementations, pretrained weights, latent frame rates, dimensions, encoder/decoder architectures, and model I/O in one place.

🔊 Audio / Sound

General audio, environmental sound, Foley, and multi-domain reconstruction. 27 models · 18 with weights.

Model and resources	Year–Month	Sample rate	Frame rate	Dim	Feature	Architecture
MiniMax H3 AudioVAE Code · Weights	2026-07	32 kHz	40 Hz	32	stereo waveform	CNN + Transformer
OmniVAE audio-only Paper · Code · Weights	2026-07	48 kHz	50 Hz	128	waveform	CNN
Qwen-Audio-3 Shared VAE Paper	2026-07	48 kHz	25 Hz	128	stereo waveform	CNN
Qwen-Audio-VAE Paper	2026-07	24 kHz	12.5 Hz	128	waveform	CNN + Transformer
STFT-VAE Code · Weights	2026-07	24 kHz	3.125 Hz	128	complex STFT	Transformer
AudioCALM Audio VAE Paper	2026-06	44.1 kHz	≈10.75 Hz	64	stereo waveform	CNN + Transformer
KVAE-Audio Code · Weights	2026-06	48 kHz	50 Hz	64	waveform	CNN
STAR-VAE Paper	2026-06	N/R	N/R	N/R	waveform	N/R
UniSonate Mel-VAE Paper	2026-04	44.1 kHz	43 Hz	40	mel spectrogram	CNN
GenAE Paper	2026-02	44.1 kHz	13.125 Hz	N/R	multichannel waveform	CNN + Transformer
Ming-omni-tts continuous tokenizer Code · Weights	2026-02	44.1 kHz	12.5 Hz	64	waveform	Transformer
LTX-2 Audio VAE Code · Weights	2026-01	16 kHz	25 Hz	128	mel spectrogram	CNN
Omni2Sound OOB/Wav VAE Paper · Code · Weights	2026-01	16 kHz	25 Hz	64	waveform	CNN
HunyuanVideo-Foley Audio VAE Paper · Code · Weights	2025-08	48 kHz	50 Hz	128	waveform	CNN
Kling-Foley Mel-VAE Paper	2025-06	44.1 kHz	43 Hz	40	mel spectrogram	CNN
MMAudio VAE 16 kHz Paper · Code · Weights	2024-12	16 kHz	31.25 Hz	20	mel spectrogram	CNN
MMAudio VAE 44.1 kHz Paper · Code · Weights	2024-12	44.1 kHz	43.07 Hz	40	mel spectrogram	CNN
MultiFoley DAC-VAE Paper	2024-11	48 kHz	40 Hz	64	waveform	CNN
DACVAE / Movie Gen Audio Autoencoder Paper · Code · Weights	2024-10	48 kHz	25 Hz	128	waveform	CNN
EzAudio 1-D VAE Paper · Code · Weights	2024-09	24 kHz	50 Hz	128	waveform	CNN
StableAudio1.0 / Stable Audio Open VAE Paper · Code · Weights	2024-07	44.1 kHz	21.5 Hz	64	waveform	CNN
Wave-VAE (Stability AI) Code	2024-07	44.1 kHz	21 Hz	64	waveform	N/R
Auffusion VAE Paper · Code · Weights	2024-01	16 kHz	12.5 Hz	128	mel-spectrogram image	CNN
AudioLDM 2 VAE Paper · Code · Weights	2023-08	16 kHz	25 Hz	128	mel spectrogram	CNN
Make-An-Audio 2 Audio VAE Paper · Code · Weights	2023-05	16 kHz	31.25 Hz	20	mel spectrogram	CNN + Transformer
AudioLDM VAE Paper · Code · Weights	2023-01	16 kHz	25 Hz	128	mel spectrogram	CNN
Make-An-Audio VAE Paper · Code · Weights	2023-01	16 kHz	7.8 Hz	40	mel spectrogram	CNN

↑ Back to top

🎵 Music

Autoencoders designed for music generation and representation. 13 models · 10 with weights.

Model and resources	Year–Month	Sample rate	Frame rate	Dim	Feature	Architecture
SAME-L Paper · Code · Weights	2026-05	44.1 kHz	10.77 Hz	256	waveform	Transformer
SAME-S Paper · Code · Weights	2026-05	44.1 kHz	10.77 Hz	256	waveform	Transformer
ACE-Step 1.5 VAE Paper · Code · Weights	2026-02	48 kHz	25 Hz	64	waveform	CNN
DiffRhythm 2 Music VAE Paper · Code	2025-10	24 kHz	5 Hz	64	waveform	CNN + Transformer
CoDiCodec continuous branch Paper · Code · Weights	2025-09	44.1 kHz	10.77 Hz	64	complex STFT	CNN + Transformer
epsilonar-VAE Paper · Code	2025-09	44.1 kHz	43 Hz	N/R	stereo waveform	CNN + Transformer
ACE-Step DCAE Paper · Code · Weights	2025-06	32 kHz	10.77 Hz	128	mel spectrogram	CNN
DiffRhythm VAE Paper · Code · Weights	2025-03	44.1 kHz	21.53 Hz	64	waveform	CNN
Music2Latent Paper · Code · Weights	2024-08	44.1 kHz	10.77 Hz	64	complex STFT	CNN + Transformer
Descript Audio VAE (community) Code · Weights	2024-01	44.1 kHz	86.13 Hz	128	waveform	CNN
Moûsai Diffusion Autoencoder Paper · Code	2023-01	48 kHz	N/R	32	magnitude spectrogram	CNN
Musika autoencoder Paper · Code · Weights	2022-08	22.05 kHz	11.89 / 23.78 Hz	32 / 64	magnitude spectrogram	CNN
RAVE v2 (MusicNet) Paper · Code · Weights	2021-11	44.1 kHz	21.53 Hz	16	waveform	CNN

↑ Back to top

🎤 Singing

Singing reconstruction, singing voice synthesis, and voice conversion. 5 models · 1 with weights.

Model and resources	Year–Month	Sample rate	Frame rate	Dim	Feature	Architecture
FM-Singer Paper · Code · Weights	2026-01	44.1 kHz	86.13 Hz	192	score/lyrics + waveform	CNN
CSSinger Paper	2024-12	44.1 kHz	86.13 Hz	192	score/lyrics + waveform	CNN
UniSyn Paper	2022-12	24 kHz	80 Hz	N/R	score/lyrics + waveform	CNN
VISinger 2 Paper	2022-11	44.1 kHz	N/R	192	score/lyrics + waveform	CNN
VISinger Paper	2021-10	24 kHz	N/R	N/R	score/lyrics + waveform	CNN

↑ Back to top

🗣️ Speech

Speech reconstruction and conditional text-to-speech VAEs. 13 models · 11 with weights.

Model and resources	Year–Month	Sample rate	Frame rate	Dim	Feature	Architecture
RedAE (FireRedAudio) Paper · Code · Weights	2026-08	24 kHz	25 Hz	64	waveform	Transformer + iSTFT
dots.tts AudioVAE Paper · Code · Weights	2026-06	48 kHz	25 Hz	128	waveform	CNN
VoxCPM2 AudioVAE Paper · Code · Weights	2026-06	16 kHz	25 Hz	64	waveform	CNN
HoliTok Paper · Code · Weights	2026-05	48 kHz	25 Hz	128	waveform	CNN
LongCat Wav-VAE Paper · Code · Weights	2026-03	24 kHz	11.72 Hz	64	waveform	CNN
MingTok-Audio Paper · Code · Weights	2025-11	16 kHz	50 Hz	64	waveform	Transformer
SALAD-VAE Paper	2025-10	N/R	7.8 Hz	64 / 128	complex STFT	CNN
Semantic-VAE Paper · Code · Weights	2025-09	16 kHz	40 Hz	64	condition + waveform	CNN
Semantic-VAE-600k Paper · Code	2025-09	16 kHz	40 Hz	64	condition + waveform	CNN
Semantic-VAE-A16 Paper · Code · Weights	2025-09	16 kHz	40 Hz	16	condition + waveform	CNN
Semantic-VAE-A64 Paper · Code · Weights	2025-09	16 kHz	40 Hz	64	condition + waveform	CNN
VoxCPM AudioVAE Paper · Code · Weights	2025-09	16 kHz	25 Hz	64	waveform	CNN
VibeVoice Acoustic Tokenizer Paper · Code · Weights	2025-08	24 kHz	7.5 Hz	64	waveform	CNN

↑ Back to top

Similar repos

kadirnar/awesome-codec-architectures

Visual catalog of neural audio and speech codec architectures, audio VAEs and continuous…

kandinskylab/KVAE-Audio

Model

KVAE-Audio is a continuous, full-band (48 kHz) audio autoencoder. It compresses raw waveforms into…

earlab/EAR_VAE2

Model

εar-VAE2

Eps-Acoustic-Revolution-Lab/EAR_VAE2

Official implementation for paper "Fourier is Frontier: Frequency-Aware Autoencoding for…

m-a-p/YuE2-Vae

Model

stabilityai/SAME-L

Model

SAME: A Semantically-Aligned Music Autoencoder

affige/genmusic_demo_list

a list of demo websites for automatic music generation research

799

fluxions/stftvae

Model

STFT-VAE

WWWWxp/awesome-continuous-audio-vae

A curated catalog of continuous audio VAEs and autoencoders for audio, music, singing, and speech.

0 commits

updated Sep 4, 2026

See the code

README

🎧 Awesome Continuous Audio VAE

A curated catalog of continuous audio VAEs and autoencoders for audio, music, singing, and speech.

Audio / Sound · Music · Singing · Speech

Papers, official implementations, pretrained weights, latent frame rates, dimensions, encoder/decoder architectures, and model I/O in one place.

🔊 Audio / Sound

General audio, environmental sound, Foley, and multi-domain reconstruction. 27 models · 18 with weights.

Model and resources	Year–Month	Sample rate	Frame rate	Dim	Feature	Architecture
MiniMax H3 AudioVAE Code · Weights	2026-07	32 kHz	40 Hz	32	stereo waveform	CNN + Transformer
OmniVAE audio-only Paper · Code · Weights	2026-07	48 kHz	50 Hz	128	waveform	CNN
Qwen-Audio-3 Shared VAE Paper	2026-07	48 kHz	25 Hz	128	stereo waveform	CNN
Qwen-Audio-VAE Paper	2026-07	24 kHz	12.5 Hz	128	waveform	CNN + Transformer
STFT-VAE Code · Weights	2026-07	24 kHz	3.125 Hz	128	complex STFT	Transformer
AudioCALM Audio VAE Paper	2026-06	44.1 kHz	≈10.75 Hz	64	stereo waveform	CNN + Transformer
KVAE-Audio Code · Weights	2026-06	48 kHz	50 Hz	64	waveform	CNN
STAR-VAE Paper	2026-06	N/R	N/R	N/R	waveform	N/R
UniSonate Mel-VAE Paper	2026-04	44.1 kHz	43 Hz	40	mel spectrogram	CNN
GenAE Paper	2026-02	44.1 kHz	13.125 Hz	N/R	multichannel waveform	CNN + Transformer
Ming-omni-tts continuous tokenizer Code · Weights	2026-02	44.1 kHz	12.5 Hz	64	waveform	Transformer
LTX-2 Audio VAE Code · Weights	2026-01	16 kHz	25 Hz	128	mel spectrogram	CNN
Omni2Sound OOB/Wav VAE Paper · Code · Weights	2026-01	16 kHz	25 Hz	64	waveform	CNN
HunyuanVideo-Foley Audio VAE Paper · Code · Weights	2025-08	48 kHz	50 Hz	128	waveform	CNN
Kling-Foley Mel-VAE Paper	2025-06	44.1 kHz	43 Hz	40	mel spectrogram	CNN
MMAudio VAE 16 kHz Paper · Code · Weights	2024-12	16 kHz	31.25 Hz	20	mel spectrogram	CNN
MMAudio VAE 44.1 kHz Paper · Code · Weights	2024-12	44.1 kHz	43.07 Hz	40	mel spectrogram	CNN
MultiFoley DAC-VAE Paper	2024-11	48 kHz	40 Hz	64	waveform	CNN
DACVAE / Movie Gen Audio Autoencoder Paper · Code · Weights	2024-10	48 kHz	25 Hz	128	waveform	CNN
EzAudio 1-D VAE Paper · Code · Weights	2024-09	24 kHz	50 Hz	128	waveform	CNN
StableAudio1.0 / Stable Audio Open VAE Paper · Code · Weights	2024-07	44.1 kHz	21.5 Hz	64	waveform	CNN
Wave-VAE (Stability AI) Code	2024-07	44.1 kHz	21 Hz	64	waveform	N/R
Auffusion VAE Paper · Code · Weights	2024-01	16 kHz	12.5 Hz	128	mel-spectrogram image	CNN
AudioLDM 2 VAE Paper · Code · Weights	2023-08	16 kHz	25 Hz	128	mel spectrogram	CNN
Make-An-Audio 2 Audio VAE Paper · Code · Weights	2023-05	16 kHz	31.25 Hz	20	mel spectrogram	CNN + Transformer
AudioLDM VAE Paper · Code · Weights	2023-01	16 kHz	25 Hz	128	mel spectrogram	CNN
Make-An-Audio VAE Paper · Code · Weights	2023-01	16 kHz	7.8 Hz	40	mel spectrogram	CNN

↑ Back to top

🎵 Music

Autoencoders designed for music generation and representation. 13 models · 10 with weights.

Model and resources	Year–Month	Sample rate	Frame rate	Dim	Feature	Architecture
SAME-L Paper · Code · Weights	2026-05	44.1 kHz	10.77 Hz	256	waveform	Transformer
SAME-S Paper · Code · Weights	2026-05	44.1 kHz	10.77 Hz	256	waveform	Transformer
ACE-Step 1.5 VAE Paper · Code · Weights	2026-02	48 kHz	25 Hz	64	waveform	CNN
DiffRhythm 2 Music VAE Paper · Code	2025-10	24 kHz	5 Hz	64	waveform	CNN + Transformer
CoDiCodec continuous branch Paper · Code · Weights	2025-09	44.1 kHz	10.77 Hz	64	complex STFT	CNN + Transformer
epsilonar-VAE Paper · Code	2025-09	44.1 kHz	43 Hz	N/R	stereo waveform	CNN + Transformer
ACE-Step DCAE Paper · Code · Weights	2025-06	32 kHz	10.77 Hz	128	mel spectrogram	CNN
DiffRhythm VAE Paper · Code · Weights	2025-03	44.1 kHz	21.53 Hz	64	waveform	CNN
Music2Latent Paper · Code · Weights	2024-08	44.1 kHz	10.77 Hz	64	complex STFT	CNN + Transformer
Descript Audio VAE (community) Code · Weights	2024-01	44.1 kHz	86.13 Hz	128	waveform	CNN
Moûsai Diffusion Autoencoder Paper · Code	2023-01	48 kHz	N/R	32	magnitude spectrogram	CNN
Musika autoencoder Paper · Code · Weights	2022-08	22.05 kHz	11.89 / 23.78 Hz	32 / 64	magnitude spectrogram	CNN
RAVE v2 (MusicNet) Paper · Code · Weights	2021-11	44.1 kHz	21.53 Hz	16	waveform	CNN

↑ Back to top

🎤 Singing

Singing reconstruction, singing voice synthesis, and voice conversion. 5 models · 1 with weights.

Model and resources	Year–Month	Sample rate	Frame rate	Dim	Feature	Architecture
FM-Singer Paper · Code · Weights	2026-01	44.1 kHz	86.13 Hz	192	score/lyrics + waveform	CNN
CSSinger Paper	2024-12	44.1 kHz	86.13 Hz	192	score/lyrics + waveform	CNN
UniSyn Paper	2022-12	24 kHz	80 Hz	N/R	score/lyrics + waveform	CNN
VISinger 2 Paper	2022-11	44.1 kHz	N/R	192	score/lyrics + waveform	CNN
VISinger Paper	2021-10	24 kHz	N/R	N/R	score/lyrics + waveform	CNN

↑ Back to top

🗣️ Speech

Speech reconstruction and conditional text-to-speech VAEs. 13 models · 11 with weights.

Model and resources	Year–Month	Sample rate	Frame rate	Dim	Feature	Architecture
RedAE (FireRedAudio) Paper · Code · Weights	2026-08	24 kHz	25 Hz	64	waveform	Transformer + iSTFT
dots.tts AudioVAE Paper · Code · Weights	2026-06	48 kHz	25 Hz	128	waveform	CNN
VoxCPM2 AudioVAE Paper · Code · Weights	2026-06	16 kHz	25 Hz	64	waveform	CNN
HoliTok Paper · Code · Weights	2026-05	48 kHz	25 Hz	128	waveform	CNN
LongCat Wav-VAE Paper · Code · Weights	2026-03	24 kHz	11.72 Hz	64	waveform	CNN
MingTok-Audio Paper · Code · Weights	2025-11	16 kHz	50 Hz	64	waveform	Transformer
SALAD-VAE Paper	2025-10	N/R	7.8 Hz	64 / 128	complex STFT	CNN
Semantic-VAE Paper · Code · Weights	2025-09	16 kHz	40 Hz	64	condition + waveform	CNN
Semantic-VAE-600k Paper · Code	2025-09	16 kHz	40 Hz	64	condition + waveform	CNN
Semantic-VAE-A16 Paper · Code · Weights	2025-09	16 kHz	40 Hz	16	condition + waveform	CNN
Semantic-VAE-A64 Paper · Code · Weights	2025-09	16 kHz	40 Hz	64	condition + waveform	CNN
VoxCPM AudioVAE Paper · Code · Weights	2025-09	16 kHz	25 Hz	64	waveform	CNN
VibeVoice Acoustic Tokenizer Paper · Code · Weights	2025-08	24 kHz	7.5 Hz	64	waveform	CNN