FireRedAudio

Demo of FireRedAudio — a general-purpose audio language model with decoupled continuous representations for both understanding and generation (paper, code, samples).

A single 9B backbone drives four tabs:

How it works

Understanding and generation use different continuous representations:

• Understanding — a Whisper-style 32-layer audio encoder (128-mel, 16 kHz) produces 12.5 Hz features that are spliced into the LLM's token stream.
• Generation — a RedAE continuous audio VAE gives 25 Hz / 64-dim latents at 24 kHz. The LLM autoregressively predicts a hidden state per group of 4 latents, and a small RedDiT flow-matching head (11 layers, 10 denoising steps, CFG 2.0) turns each hidden state into the actual latents. A separate RedAE decoder (Qwen3 + ISTFT head) renders the 24 kHz waveform.

Because both live in one model, generation tabs can also emit text (voice design writes its own structured timbre tags; semantic editing writes the rewritten transcript) before switching to audio.

Notes

• Runs on ZeroGPU. The backbone is bf16 (~21 GB) and the RedAE decoder is fp32; the first request after a cold start pays the weight-streaming cost.
• Voice cloning needs the reference transcript to be exact — a mismatch degrades the clone.
• flash-attn, liger-kernel, causal-conv1d and flash-linear-attention are optional upstream accelerations and are not installed here; the pure-PyTorch fallbacks change bf16 reduction order, so output differs numerically (not qualitatively) from the paper's setup.
• This Space wraps the upstream fireredaudio/ package and inference.py verbatim (Apache-2.0), so prompts and sampling parameters match the reference implementation.
• Example audio in assets/ comes from the authors' own demo page and the upstream repository (Apache-2.0), and is derived from FLEURS, Common Voice and MMAU test material.