qwen3-tts-rs (fork)

Fork of TrevorS/qwen3-tts-rs with modifications for concurrent batched inference in concurrent-faster-qwen3-server.

Pure Rust inference for Qwen3-TTS, a text-to-speech model from Alibaba. Built on candle.

Modifications from upstream

• synthesize_batch() / synthesize_batch_with_voices() — batched autoregressive generation (N sequences per GPU forward pass)
• generate_acoustic_codes_batched() — batched code predictor
• QuantizedKVCache (U8) — experimental, disabled by default
• get_codec_embedding_batch() — shape fix (unsqueeze(0) vs unsqueeze(1)) for ICL voice cloning
• ICL warm-up frame skip (ref_text_len - 3) to remove ref_text audio from output
• ICL KV cache dynamic allocation for long reference text
• Pre-allocated shared tensors in prefill phase
• unsafe impl Send+Sync for Qwen3TTS for Arc sharing across threads

License

Apache-2.0 (same as upstream). Original work by TrevorS.

---

Original README below:

Changelog (fork)

See CHANGELOG.md for detailed modifications from upstream.

Summary of changes

• Batched autoregressive generation (N sequences per GPU forward pass)
• Batched code predictor and vocoder decoding
• Voice cloning in batch mode (mixed batches)
• ICL voice clone shape fix + warm-up frame skip
• QuantizedKVCache U8 (experimental)
• Adaptive max_length for call center text
• Arc<Qwen3TTS> thread safety

Upstream changelog (TrevorS/qwen3-tts-rs)

Click to expand

0.4.0

• Pre-allocated KV cache with InplaceOp2 (zero-copy CUDA writes, no Tensor::cat)
• GPU-side repetition penalty mask (incremental slice_assign, eliminates growing CPU transfer)
• Deferred acoustic codes transfer (single bulk GPU→CPU at end of generation)
• Fused residual + RMSNorm CUDA kernel
• GPU→CPU syncs reduced from 3/frame to 1/frame (4-byte EOS check)
• Non-streaming RTF: 0.48–0.67 across all variants (97-100% of theoretical throughput)

0.3.0

• GPU-side sampling: batched argmax, on-device top-k/top-p/repetition penalty
• Eliminated 15 of 16 GPU→CPU syncs per frame in code predictor
• Cached token suppression mask in streaming sessions
• Tokenizer fallback from vocab.json + merges.txt when tokenizer.json is unavailable
• Profiling infrastructure: Chrome tracing, flamegraph, Nsight Systems via Makefile
• Benchmarked all 4 model variants (0.6B Base, 1.7B Base/CustomVoice/VoiceDesign)

0.2.0

• ICL voice cloning now works correctly with proper reference audio
• Fixed WAV output format (WAVEX/float32 → standard WAV/PCM16)
• Improved tokenizer path resolution with explicit --tokenizer-dir override
• Added benchmarking suite (Criterion micro-benchmarks + E2E speed tests)
• Automatic resampling of reference audio to 24kHz for voice cloning

Acknowledgements

• Qwen Team (Alibaba) — Qwen3-TTS model, weights, and technical report
• candle — Rust ML framework by Hugging Face
• mlx-audio — reference implementation that helped clarify model details
• Claude Code — wrote the code

Features

Added in this fork

• Batched autoregressive generation — N sequences per GPU forward pass (16.59x RT at batch=16)
• Batched vocoder decoding — single ONNX pass for all sequences (+24% throughput)
• Batched code predictor — generate_acoustic_codes_batched() for streaming and batch paths
• Voice cloning in batch mode — mixed batches (some clone, some default voice)
• ICL voice clone fixes — shape mismatch fix, warm-up frame skip, KV cache dynamic allocation
• Adaptive max_length — reduces KV cache ~75% for short call center text
• QuantizedKVCache U8 — experimental INT8 KV cache (disabled by default)
• Thread-safe model — Arc<Qwen3TTS> via unsafe impl Send+Sync for concurrent servers

From upstream (TrevorS/qwen3-tts-rs)

• CPU inference with optional MKL/Accelerate for faster BLAS operations
• CUDA support for NVIDIA GPU acceleration with bf16 and Flash Attention 2
• Metal support for Apple Silicon
• Streaming synthesis for low-latency audio output
• Voice cloning via x-vector or ICL (in-context learning) from reference audio (Base models)
• Preset speakers with 9 built-in voices (CustomVoice models)
• Text-described voices via natural language prompts (VoiceDesign models)
• Auto-detection of model variant from config.json
• HuggingFace Hub integration for easy model downloads

Performance

Benchmarked on an NVIDIA DGX Spark (GB10 Blackwell, ARM Cortex-X925, 120 GB unified memory). Default generation parameters, seed 42, 2 warmup + 3 timed iterations.

RTF (real-time factor) = wall-clock / audio duration. < 1.0 is faster than real-time. Non-streaming results shown above. Streaming adds ~8-12% overhead with TTFA ~444 ms (0.6B) / ~580 ms (1.7B).

See docs/BENCHMARKS.md for full results, test corpus, micro-benchmarks, and reproduction instructions.

Samples

All samples generated with 1.7B models, seed 42. Text: "The sun set behind the mountains, painting the sky in shades of gold and violet."

CustomVoice — Ryan

🔊 Listen

CustomVoice — Serena

🔊 Listen

Voice Clone — ICL

🔊 Listen

VoiceDesign — Radio Announcer

🔊 Listen

VoiceDesign — Storyteller

🔊 Listen

VoiceDesign — Sportscaster

🔊 Listen

Model Variants

Five official model variants exist across two size classes. Each variant supports a different speaker conditioning method:

Which model should I use?

• Want to clone a specific voice? Use a Base model with --ref-audio (ICL mode) or --ref-audio --x-vector-only (faster, lower quality).
• Want a quick preset voice? Use a CustomVoice model with --speaker.
• Want to describe a voice in text? Use 1.7B VoiceDesign with --instruct.
• Unsure? Start with 0.6B CustomVoice for the fastest results.

Valid combinations

Using the wrong combination (e.g. preset speakers on a Base model) won't crash, but produces unpredictable voice output. The library and CLI warn when this happens.

Installation

Add to your Cargo.toml:

[dependencies]
qwen3-tts = { version = "0.1", features = ["hub"] }

Feature Flags

Quick Start

Preset speakers (CustomVoice)

use qwen3_tts::{Qwen3TTS, Speaker, Language, auto_device};

fn main() -> anyhow::Result<()> {
    let device = auto_device()?;
    let model = Qwen3TTS::from_pretrained("path/to/customvoice_model", device)?;

    let audio = model.synthesize_with_voice(
        "Hello, world!",
        Speaker::Ryan,
        Language::English,
        None,
    )?;
    audio.save("output.wav")?;
    Ok(())
}

Available speakers: Serena, Vivian, UncleFu, Ryan, Aiden, OnoAnna, Sohee, Eric, Dylan

Voice cloning (Base)

use qwen3_tts::{Qwen3TTS, Language, AudioBuffer, auto_device};

fn main() -> anyhow::Result<()> {
    let device = auto_device()?;
    let model = Qwen3TTS::from_pretrained("path/to/base_model", device)?;

    // Load reference audio
    let ref_audio = AudioBuffer::load("reference_voice.wav")?;

    // ICL mode: full voice cloning with reference text
    let prompt = model.create_voice_clone_prompt(&ref_audio, Some("transcript of ref audio"))?;

    // x_vector_only: faster, speaker embedding only (no reference text needed)
    // let prompt = model.create_voice_clone_prompt(&ref_audio, None)?;

    let audio = model.synthesize_voice_clone(
        "Hello in the cloned voice!",
        &prompt,
        Language::English,
        None,
    )?;
    audio.save("cloned.wav")?;
    Ok(())
}

Text-described voice (VoiceDesign)

use qwen3_tts::{Qwen3TTS, Language, auto_device};

fn main() -> anyhow::Result<()> {
    let device = auto_device()?;
    let model = Qwen3TTS::from_pretrained("path/to/voicedesign_model", device)?;

    let audio = model.synthesize_voice_design(
        "Hello from a designed voice!",
        "A cheerful young female voice with high pitch and energetic tone",
        Language::English,
        None,
    )?;
    audio.save("designed.wav")?;
    Ok(())
}

With custom options

use qwen3_tts::{Qwen3TTS, SynthesisOptions, auto_device};

fn main() -> anyhow::Result<()> {
    let device = auto_device()?;
    let model = Qwen3TTS::from_pretrained("path/to/model", device)?;

    let options = SynthesisOptions {
        temperature: 0.8,
        top_k: 30,
        top_p: 0.85,
        repetition_penalty: 1.05,
        ..Default::default()
    };
    let audio = model.synthesize("Custom settings!", Some(options))?;
    audio.save("output.wav")?;
    Ok(())
}

Streaming synthesis

For low-latency applications, stream audio in chunks:

use qwen3_tts::{Qwen3TTS, Speaker, Language, SynthesisOptions, auto_device};

fn main() -> anyhow::Result<()> {
    let device = auto_device()?;
    let model = Qwen3TTS::from_pretrained("path/to/model", device)?;

    let options = SynthesisOptions {
        chunk_frames: 10, // ~800ms per chunk
        ..Default::default()
    };

    for chunk in model.synthesize_streaming(
        "Hello, world!",
        Speaker::Ryan,
        Language::English,
        options,
    )? {
        let audio = chunk?;
        // Play or stream this chunk
        println!("Got {} samples", audio.samples.len());
    }
    Ok(())
}

With HuggingFace Hub

use qwen3_tts::{Qwen3TTS, ModelPaths, auto_device};

fn main() -> anyhow::Result<()> {
    let paths = ModelPaths::download(None)?;
    let device = auto_device()?;

    let model = Qwen3TTS::from_paths(&paths, device)?;
    let audio = model.synthesize("Hello from HuggingFace!", None)?;
    audio.save("output.wav")?;
    Ok(())
}

Architecture

The TTS pipeline consists of three stages:

1. TalkerModel: 28-layer transformer generating semantic tokens from text autoregressively. Uses MRoPE (multimodal rotary position encoding) across all variants.

2. CodePredictor: 5-layer decoder that generates 15 acoustic tokens per semantic token. Always 1024 hidden dim; 1.7B models use a projection layer to bridge from the talker's 2048-dim space.

3. Decoder12Hz: Converts 16-codebook tokens to 24kHz audio via ConvNeXt blocks and transposed convolution upsampling. Shared across all model variants.

Text --> TalkerModel --> Semantic Token --> CodePredictor --> [16 codes] --> Decoder --> Audio
              ^                                  ^
         (autoregressive,                  (per frame,
          one per frame)                    15 acoustic codes)

CLI

The model variant is auto-detected from config.json. The CLI warns if your flags don't match the model type.

# CustomVoice: preset speaker
cargo run --release --features cli --bin generate_audio -- \
  --model-dir path/to/customvoice \
  --text "Hello world" \
  --speaker ryan \
  --language english \

# Base: voice cloning (ICL — best quality, requires reference text)
cargo run --release --features cli --bin generate_audio -- \
  --model-dir path/to/base \
  --text "Hello world" \
  --ref-audio reference.wav \
  --ref-text "transcript of the reference audio"

# Base: voice cloning (x_vector_only — faster, no transcript needed)
cargo run --release --features cli --bin generate_audio -- \
  --model-dir path/to/base \
  --text "Hello world" \
  --ref-audio reference.wav \
  --x-vector-only

# VoiceDesign: describe the voice you want
cargo run --release --features cli --bin generate_audio -- \
  --model-dir path/to/voicedesign \
  --text "Hello world" \
  --instruct "A cheerful young female voice with high pitch and energetic tone" \
  --language english

# Reproducible generation with fixed seed
cargo run --release --features cli --bin generate_audio -- \
  --model-dir path/to/model \
  --text "Hello" \
  --seed 42

CLI options

GPU Acceleration

On CUDA devices, the talker and code predictor automatically run in bf16 for lower memory usage and faster inference. The codec decoder and speaker encoder remain in f32 (convolutional, no attention).

For best performance, build with Flash Attention 2 (requires CUDA toolkit in PATH):

cargo build --release --features flash-attn,cli

Docker

For GPU builds, use the build script which auto-detects your GPU architecture:

# Build GPU image (auto-detects compute capability)
./build-docker.sh qwen3-tts flash-attn,cli

# Build CPU-only image
./build-docker.sh qwen3-tts-cpu cli

The script builds inside a running container with GPU access, ensuring correct PTX compilation for your exact hardware (Ampere, Ada, Hopper, Blackwell, etc.).

Run inference:

docker run --gpus all \
  -v /path/to/models:/models \
  -v /path/to/output:/output \
  qwen3-tts \
    --model-dir /models/0.6b-customvoice \
    --speaker ryan \
    --text "Hello world, this is a test." \
    --device cuda \
    --output /output/hello.wav

For CPU-only builds without GPU access, use the Dockerfile directly:

docker build --build-arg FEATURES=cli --build-arg BASE=ubuntu:22.04 -t qwen3-tts-cpu .

Dtype behavior

Model Files

All models share the same speech tokenizer and text tokenizer.

Supported languages

English, Chinese, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian

Sample Rate

Output audio is always 24kHz mono. Use audio::resample() for other rates:

use qwen3_tts::audio;

let audio_48k = audio::resample(&audio, 48000)?;

License

MIT License. See the main Qwen3-TTS repository for model license information.