STFT-VAE

Continuous (VAE) neural audio codec built on the STFT. Audio is transformed to a complex spectrogram, a transformer encoder downsamples it to a low-rate continuous latent, and a transformer decoder + ISTFT reconstruct the waveform — phase is carried by the STFT/ISTFT rather than a learned vocoder.

• Sample rate: 24 kHz
• Latent: 3.125 Hz frame rate, 128-dim continuous (no quantization)
• Params: ~116M
• Files: config.json (architecture) + model.safetensors (fp32 weights)

Usage

pip install "stftvae[cli]"

import torch
from stftvae import STFTVAE

vae = STFTVAE.from_pretrained("fluxions/stftvae", device="cuda")

audio = torch.randn(1, 24000)          # (1, T) mono @ 24 kHz
recons = vae.reconstruct(audio)        # (1, 1, T)

# or encode / decode separately
latent, length = vae.encode(audio)     # (1, 128, T_latent)
recons = vae.decode(latent, length=length)

Command line:

python -m stftvae input.wav output.wav --model fluxions/stftvae

The encoder returns the deterministic posterior mean (no sampling noise at inference). Code: https://github.com/fluxions-ai/stftvae

License

MIT