---

OmniVoice Rust Port
GPU-first Rust workspace for OmniVoice inference, parity validation, and CLI execution with Candle.

📚 Table of Contents

• What is this?
• Demo
• Listening samples
• Key Features
• Installation & Setup
• How to Start Using
• System Requirements
• Acknowledgments
• License

✨ What is this?

This repository is a Rust workspace focused on porting OmniVoice inference to Candle with GPU-first execution.

It contains:

• crates/omnivoice-infer — the two-stage inference pipeline
• crates/omnivoice-cli — CLI for prompt prep, stage0/stage1 inspection, and full inference
• crates/omnivoice-server — separate OpenAI-compatible HTTP speech server binary
• docs/contracts — phase-by-phase behavior contracts
• tools — local reference and support scripts

The project prefers CUDA and Metal when available, but full OmniVoice inference is available on CPU as well as GPU backends.

🎬 Demo

There is no public GUI demo in this workspace. The primary interface is the CLI.

cargo run -p omnivoice-cli --features cuda -- infer `
  --text "Hello, this is a test of zero-shot text-to-speech." `
  --language en `
  --output artifacts\demo.wav `
  --device cuda:0 `
  --seed 1234

If you want to force a local bundle or a specific Hugging Face repo, pass --model:

cargo run -p omnivoice-cli --features cuda -- infer `
  --model model `
  --text "Hello from a local model bundle." `
  --output artifacts\demo-local.wav

For an OpenAI-compatible speech endpoint, run the separate server binary:

$env:OMNIVOICE_API_KEY="local-dev-token"
cargo run -p omnivoice-server --features cuda -- `
  --host 127.0.0.1 `
  --port 8000 `
  --device cuda:0

🎧 Listening samples

Pre-generated WAVs let you A/B the upstream Python OmniVoice reference against this Rust port on the same prompts and seed.

Quick CUDA pairs (open both links and listen):

Full 15-scenario CUDA table and CPU folders are in the comparison index.

🚀 Key Features

• Two-stage OmniVoice inference pipeline in Rust
• GPU-first runtime selection: CUDA -> Metal -> CPU fallback
• Full inference paths available on CPU and GPU
• Stage0 and Stage1 parity harnesses against official OmniVoice references
• Voice clone, voice design, auto voice, batch, and long-form chunked inference
• CLI workflows for prepare-prompt, stage0-debug, stage1-decode, and infer
• Separate omnivoice-server binary with /, /health, /v1/models, and /v1/audio/speech
• Server-side readiness reporting, base-path mounting, CORS/OPTIONS, and JSON or multipart request support for TTS

Hardware Acceleration

🛠️ Installation & Setup

Prerequisites

• Rust toolchain
• CPU-only inference works; GPU acceleration is optional
• For CUDA: NVIDIA GPU and compatible driver/toolkit
• For Metal: macOS with Metal support
• Either a local OmniVoice bundle or network access to download models into the Hugging Face cache
• Official upstream reference materials available locally

Development

cargo fmt
cargo test -p omnivoice-infer --features cuda --test phase_status -- --nocapture --test-threads=1
cargo test -p omnivoice-cli --features cuda --test phase10_cli_cuda -- --nocapture --test-threads=1

Quality Checks

cargo fmt
cargo clippy --workspace --all-targets
cargo test -p omnivoice-infer --features cuda --test phase10_cuda_acceptance -- --nocapture --test-threads=1
cargo test -p omnivoice-cli --features cuda --test phase10_cli_cuda -- --nocapture --test-threads=1

📖 How to Start Using

1. Either pass --model <local-path> or let the CLI/server auto-resolve k2-fsa/OmniVoice from Hugging Face.
2. If you omit --asr-model, Whisper defaults to oxide-lab/whisper-large-v3-turbo-GGUF and downloads the Candle-compatible config.json, tokenizer.json, and whisper-large-v3-turbo-q4_0.gguf.
3. Keep official upstream reference materials available locally for behavior verification.
4. Run sequential GPU tests instead of launching everything at once.
5. Use omnivoice-cli infer for end-to-end synthesis.

🖥️ System Requirements

• Windows, macOS, or Linux
• Enough RAM and VRAM for OmniVoice weights and runtime tensors
• For GPU acceleration:
  • NVIDIA GPU for CUDA
  • Apple Silicon / supported macOS GPU for Metal

🙏 Acknowledgments

This workspace builds on top of the official upstream projects:

• OmniVoice
• candle
• mistral.rs

Official upstream projects remain the source of truth for behavior and engineering reference, but they are not tracked as part of this repository.

See THIRD_PARTY_LICENSES.md for focused upstream attribution relevant to this workspace.

📄 License

Apache-2.0 — see LICENSE

Copyright (c) 2026 FerrisMind