omnivoice-rs

A Rust implementation of OmniVoice TTS inference using the Candle ML framework. Generates speech from text with voice cloning, voice design, and automatic speech recognition support.

Features

• Voice cloning -- clone a voice from a reference audio file
• Voice design -- describe the desired voice style (gender, age, pitch, accent)
• Auto voice -- let the model pick a voice automatically
• Whisper ASR -- auto-transcribe reference audio when --ref-text is omitted
• Chunked generation -- long texts are split at sentence boundaries and cross-faded
• GPU acceleration -- Metal (macOS) and CUDA support
• 600+ languages -- language codes resolved via ISO 639

Installation

git clone https://github.com/k2-fsa/OmniVoice.git
cd OmniVoice/omnivoice-rs

# CPU
cargo build --release

# macOS GPU (Metal)
cargo build --release --features metal

# NVIDIA GPU (CUDA)
cargo build --release --features cuda

Quick Start

Models are downloaded automatically from HuggingFace on first run.

Voice design (no reference audio)

omnivoice-rs \
    --text "Hello, this is a test of the OmniVoice text to speech system." \
    --output output.wav

Voice design with style

omnivoice-rs \
    --text "The quick brown fox jumps over the lazy dog." \
    --instruct "female, british accent, high pitch" \
    --language en \
    --output styled.wav

Voice cloning

omnivoice-rs \
    --text "This sentence will be spoken in the cloned voice." \
    --ref-audio reference.wav \
    --ref-text "Transcript of the reference audio." \
    --language en \
    --output cloned.wav

Voice cloning with auto-transcription

When --ref-text is omitted, Whisper automatically transcribes the reference audio:

omnivoice-rs \
    --text "This sentence will be spoken in the cloned voice." \
    --ref-audio reference.wav \
    --language en \
    --output cloned.wav

The Whisper model (openai/whisper-large-v3-turbo by default) is downloaded only when needed. Use --asr-model to specify a different model.

CLI Reference

omnivoice-rs [OPTIONS] --text <TEXT> --output <OUTPUT>

Required

Voice mode

Generation parameters

System

Voice Design Instructions

The --instruct flag accepts comma-separated attributes:

Architecture

The model runs on GPU (Metal/CUDA) in FP16, while the audio tokenizer and Whisper always run on CPU in FP32 for numerical stability.

Project Structure

src/
  main.rs                        CLI entry point
  config.rs                      Config deserialization
  models/
    omnivoice.rs                 Core model + iterative generation
    qwen3_bidirectional.rs       Qwen3 backbone (bidirectional)
    dac.rs                       DAC encoder/decoder
    hubert.rs                    HuBERT feature extractor
    higgs_audio_v2.rs            Audio tokenizer (encode/decode)
    rvq.rs                       Residual vector quantization
    semantic_codec.rs            Semantic encoder/decoder
    whisper_transcribe.rs        Whisper ASR integration
  utils/
    audio.rs                     WAV I/O, resampling, silence removal
    text.rs                      Text chunking, punctuation
    duration.rs                  Duration estimation
    sampling.rs                  Gumbel sampling, top-k filtering
    voice_design.rs              Instruct validation

License

Apache-2.0