tts-rs

Narrate anything in a cloned voice, on your own Mac, offline. A 1612-word chapter becomes 11 minutes of speech in 2m 40s. One binary, no service, no API key.

▶ Hear it first: one chapter, three engines, two cloned voices

Requirements: macOS on Apple silicon. The custom kernels are Metal. Every number here was measured on an M4 with 16 GB. It builds and runs elsewhere with --no-default-features, on unit-tested CPU fallbacks. That path is a portability guarantee, not a deployment target. audio8 measures RTF 2.151 on CPU against 0.554 on Metal. Its codec suffers most: 1.235 against 0.214.

Speak your first sentence

./scripts/bootstrap.sh qwen3tts     # toolchain, checkpoint, assets, build. ~4 GB

cargo run -p tts-cli --release -- speak \
    --engine qwen3tts --voice voices/cosy-default-qwen3tts \
    --text "Hello from a fresh checkout." --out hello.wav

Start with qwen3tts. It is the best quality of the three and more than twice as fast on book-length text. The demo is there so you can disagree before you download it.

bootstrap.sh downloads and converts the checkpoint, fetches the fixtures for its gates, and builds. Nothing is manual. Add the other engines whenever you want them. All three is ~13 GB, one is ~4 GB:

./scripts/bootstrap.sh --list              # the ids, their models, what each costs
./scripts/bootstrap.sh audio8 cosyvoice    # 44.1 kHz output, and the widest language coverage
./scripts/bootstrap.sh                     # all three

Every step is skipped if its output exists, so re-running is cheap. Details in docs/reference.md.

Speak in someone's voice

A voice is a directory, and the repo ships several. Building your own takes a clip of about ten seconds and its exact transcript:

references/qwen3tts/.venv/bin/python references/qwen3tts/export_voice.py \
    --model references/qwen3tts/weights --audio clip.wav \
    --text "the exact words spoken in the clip" \
    --name my-voice --out voices/my-voice

Pass it to any command as --voice voices/my-voice. tts voice voices/my-voice prints what an asset holds without synthesising.

This export is the only step that wants Python, and it runs once per voice. Install references/<engine>/requirements.txt in a venv first. The speaker encoders stay there: the runtime loads the exported conditioning and never carries an encoder.

The engines, and what they cost

All three narrate the same 1612-word chapter (examples/chapter.txt, in the repo). Each runs in the configuration it ships in, in two cloned voices, on one M4 with 16 GB and no Python running. GitHub cannot embed audio in markdown, so the demo page plays all six in place.

That bottom row is the point of the project. A chapter becomes 11 minutes of speech in under 3 minutes, on a laptop. A 16-hour book costs about 4 hours of compute rather than 12.

Compare the wall-time column, not just RTF. The three do not produce the same duration from the same text. cosyvoice speaks slowest, 12:48 against audio8's 11:34. RTF divides by audio produced, so a slower-speaking engine flatters its own RTF.

cargo run -p tts-cli --release -- speak --engine qwen3tts \
    --voice voices/cosy-default-qwen3tts --quant f16 \
    --text-file examples/chapter.txt --out chapter.wav

qwen3tts gets there by batching across sections. That needs --quant f16 and it needs length: on a 7-segment passage it is the slowest of the three at 0.665. The other two do not batch meaningfully and are steady at any length. audio8 is 2.36× its PyTorch reference like for like, with that reference running on MPS too.

Short-passage figures, for comparison. examples/senior.txt, 132 words, median of five with the engines interleaved: audio8 0.554, cosyvoice 0.726, qwen3tts 0.665.

Narrate a whole book

scripts/narrate-book.sh --book path/to/document --out narration --engine qwen3tts
scripts/verify-narration.py narration/*.webm

Markdown in. Delivery audio and word-level timings out. One engine load for the whole run. Resumable per stage: a section with a WAV master is never re-synthesised. Deterministic under a seed. A 16-hour document costs about 4 hours of synthesis at qwen3tts's 0.260, against ~12 at cosyvoice's 0.726. Recognition adds an hour either way.

Serve it over HTTP

TTS_API_KEY=secret cargo run -p tts-serve --release -- --port 3003

curl -X POST localhost:3003/tts -H "X-API-Key: secret" \
     -H 'content-type: application/json' \
     -d '{"text":"Hello from Rust.","voice":"voices/cosy-default-male","seed":7}' \
     -o out.wav -D headers.txt

One engine, loaded once, in 3.0 s. voice and seed are per request. The first selects a voice asset without a restart. The second makes a render reproducible.

Every response carries its own cost. x-audio-seconds, x-wall-seconds, x-rtf, and x-stages with the per-stage split (llm=10.296,flow=25.129,vocoder=2.898). A client sees where the time went without a second request.

Use it as a library

use tts_core::{EngineConfig, SynthesisRequest, Voice};

let config = EngineConfig::new(tts_engines::default_root("cosyvoice"));
let engine = tts_engines::load("cosyvoice", &config)?;

let voice = Voice::load("voices/cosy-default-cosyvoice")?;
let request = SynthesisRequest::new("Hello from Rust.").with_voice(voice);
engine.validate(&request)?;                 // rejects a mismatched asset up front

let out = engine.synthesize(&request)?;
tts_core::wav::write("hello.wav", &out.audio)?;
println!("RTF {:.3}", out.stats.rtf(out.audio.seconds()));

One Engine trait. Engines are chosen by string id at request time.

Limitations

• Ten languages on qwen3tts, a closed list: en, de, es, zh, ja, fr, ko, ru, it, pt. Text outside it has no faithful path through that engine.
• A known performance regression, undiagnosed. audio8's codec and cosyvoice's vocoder are 35% and 32% slower than when first measured, while every transformer stage is unchanged. Both are convolution-heavy. The cause is likely the channels-last conv path.
• Sampled output is not reproducible across implementations. The reference draws from torch's RNG. Pass a seed for repeatability within this port. Use the greedy path if you need to compare against PyTorch.

One comparison is not worth making. cosyvoice looks 6× faster than its PyTorch reference. That is only because upstream hardcodes cuda if available else cpu and has no MPS path. Against a service that does use MPS it is ahead by about 5%.

How it is built

Three PyTorch models, ported to Rust and candle, with the Metal kernels written here. Each stage is validated against fp32 activations dumped from its reference, so a mismatch names the layer that caused it. scripts/fetch-assets.sh pulls ~130 MB of checksummed ground truth from drmhse/tts-rs-assets, so ./scripts/gates.sh runs the gates without any PyTorch installed.

Documentation

Everything else is one file: docs/reference.md.

Layout

crates/tts-core/        the Engine trait, voice assets, segmentation, WAV, the PRNG
crates/tts-nn/          shared model machinery plus the custom Metal kernels
crates/tts-engines/     the registry, the one place that knows which engines exist
crates/tts-cli/         the `tts` binary: engines / voice / speak
crates/tts-serve/       the HTTP service: one engine, loaded once, behind a semaphore
crates/tts-bench/       the thermally-honest measurement harness
crates/tts-probe/       op-level benchmarks, one binary per question
crates/{audio8,cosyvoice,qwen3tts}/   one engine each, plus its fixture gate

references/{audio8,cosyvoice,qwen3tts}/   the PyTorch side: conversion, fixtures, quality
fixtures/{audio8,cosyvoice,qwen3tts}/     per-stage ground truth the gates compare against
voices/                 voice assets, one directory each. Tracked, since they are small
examples/               senior.txt and chapter.txt, the two benchmark fixtures
scripts/                bootstrap, fetch-assets, gates, render-examples, narration

Everything shared is named tts-*. Everything engine-specific is named for its engine, and crates/audio8, crates/cosyvoice and crates/qwen3tts match the ids --engine takes. Weights and virtualenvs are not tracked; docs/reference.md builds them.