Voice Agent — a fully local voice assistant that starts responding in under 200 milliseconds, in your own voice, on open-weight models only

A real-time spoken assistant that listens, thinks, looks things up on the internet, and answers in a clone of your voice, running entirely on one laptop with no cloud inference and no proprietary models. It reacts in ~170 milliseconds (a spoken acknowledgement in your cloned voice) and speaks the first words of its answer about a second after you stop talking; the transcript is already finished before you finish pausing. It was built and measured on an Intel Core Ultra 7 155H (CPU + Arc iGPU + NPU, no NVIDIA GPU), splitting the models across the three compute units so that no stage waits for another stage's hardware.

Measured on that laptop (English, steady state, ms after you stop speaking):

Hindi / Hinglish turns take 4–6 s (multilingual STT plus Devanagari being token-expensive for the LLM). The honest framing: under 200 ms to a spoken reaction, about one second to the answer, everything local.

---

Hardware requirements

This is a compute-heavy local stack; the numbers above depend on the hardware split. What you need:

What changes without the accelerators: on CPU alone the English STT takes 0.5–0.8 s instead of 0.13 s, Whisper ~5 s, the LLM ~5 tok/s, and a cloned-voice sentence 3–4 s, so expect 3–6 s to the answer instead of ~1 s. An Intel Core Ultra (Meteor Lake or newer) gets the measured behaviour out of the box. NVIDIA GPUs and Apple Silicon run the LLM through Ollama today, but the STT/TTS accelerator backends are OpenVINO-specific, so those stages fall back to CPU until a CUDA/Metal backend is contributed (see CONTRIBUTING → Adding a backend).

---

Feature set

Conversation

• Hands-free turn taking with speculative execution: STT and LLM start on the first 64 ms of silence, ~250–320 ms before the turn is declared over; if you keep talking the work is cancelled and the partial transcript is merged into the next turn. Playback is gated on the firm end of turn, so speculation never talks over you.
• Adaptive end-of-turn: the speculative transcript decides whether a pause is a real end ("... and" waits longer).
• Cloned-voice fillers ("Hmm.", "Let me think.") pre-synthesized at startup and played within ~170 ms, only when the real answer is not ready yet; "Let me check that." when a tool call starts.
• Streaming everywhere: LLM tokens → clause chunker → TTS per clause → gapless playback; the first chunk is cut after 3 words so speech starts while the model is still writing.
• Conversation history with a per-message language tag so the model answers in the language you spoke, even after switching (English, Hindi in Devanagari, Hinglish in Latin letters).

Understanding

• English STT in ~130 ms on the Intel NPU (Granite Speech 5.0 470M TurboCTC, encoder-only, one pass).
• 99-language STT (Whisper large-v3-turbo) on the Intel GPU, used directly when the fast transcript does not look like English, verified in parallel when borderline, skipped when clearly English.
• Voice lock: a speaker-embedding model compares every utterance with your enrolled voice and ignores anyone else, in any language (your voice ~0.8 similarity, other voices < 0.3, threshold 0.45).
• Noise robustness: VAD threshold, minimum speech length, and an energy gate against the measured room noise floor; echo guard while the agent speaks; browser echo cancellation in hosted mode.

Knowledge and tools (all keyless)

• Live weather (Open-Meteo, with geocoding incl. Devanagari place names), Wikipedia summaries, DuckDuckGo search, clock, calculator, through the LLM's native tool calling.
• Speculative prefetch: weather questions are detected in the transcript (English, Hinglish, Hindi keywords) and fetched before the LLM runs, so the answer comes in one model round instead of three; forecasts cached 10 min.

Voice

• Zero-shot voice cloning from a ~20 s recording (LuxTTS, Apache-2.0), 48 kHz output, flow decoder on the GPU.
• Hindi replies spoken by a Hindi voice (Kokoro) because the clone is English-only.
• Recording window with a reading script, live level meter, take quality check (peak, clipping, silence, late start), playback, one-click activation and "test the clone"; silence is trimmed/condensed before enrollment.

Operations

• Web console: mute/unmute (Space), text test box, wav replay through the full mic path, live transcript and streamed reply, per-turn latency table and history, live settings (fillers, speculation, barge-in, end-of-turn silence, TTS steps, voice lock strictness, microphone device).
• Browser-microphone mode: the page captures your mic and plays replies itself over a WebSocket, so the console can be hosted anywhere (a static copy for Vercel is generated into web/).
• A private Ollama instance is launched with settings tuned for one low-latency conversation; nothing system-wide is modified.
• Everything is cached after the first start (OpenVINO compiled blobs, voice prompt encoding); warm start ~90 s.

---

Components: what each part does and why it exists

agent/ — the core package

tts_worker/ — the cloned-voice process

ui/ and web/ — the console

• ui/server.py: FastAPI app with a control WebSocket (events, commands, settings), the /audio WebSocket (16 kHz int16 in, 48 kHz int16 out, browser mic + playback), /record and /voice.wav, UTF-8 forced on the console (the Windows console crashed on Devanagari).
• ui/index.html: the console; ui/record.html: the voice-recording window.
• web/: static copy of the console (built by tools/build_web.py) for Vercel or any static host; it connects to whatever backend URL you enter.

main.py — CLI

Live mic mode, offline wav replay, text-only turns, voice recording, device listing, --prepare to compile and cache every model on first install.

benchmarks/, tests/, tools/

The scripts that drove every decision above: STT on CPU vs NPU vs GPU, thread sweeps, Whisper variants, Ollama time-to-first-token experiments (localhost vs 127.0.0.1, slots, prompt length), LuxTTS backends and bucket padding, vocoder conversion attempts, TTS quality checked by transcribing its own output, and an end-to-end test of the browser-audio protocol.

---

Latency engineering, in the order it mattered

1. Hardware split: STT → NPU, LLM + TTS flow → GPU, VAD/vocoder/voice lock → CPU. CPU-only stages were 0.5–4 s each; after the split every stage is 100–500 ms.
2. Speculative turn start with cancel/merge, and playback gated on the firm end.
3. 127.0.0.1, not localhost: on Windows the client tried IPv6 first and lost ~1.7 s per Ollama request.
4. One Ollama slot (private instance): prefix reuse instead of full prompt re-evaluation.
5. Short prompts: the runner saves/restores the slot's KV state around each changed prompt (~0.3 ms per token of prompt on this iGPU), so the system prompt and tool schema are kept minimal.
6. Tiny first chunk, 2 flow steps for it, 4 for the rest; fillers to cover the remaining gap.
7. Whisper only when needed, so it does not compete with the LLM for the GPU on English turns.
8. Prefetch for the most common tool (weather) and cached results.

Remaining floor on this laptop, and how to get to ~500 ms: the Ollama per-request state overhead (~250 ms) goes away by running the LLM through OpenVINO GenAI with a persistent in-process KV cache; the vocoder (100–300 ms on CPU; it does not convert to OpenVINO) needs a GPU-friendly replacement or a 1-step flow; a smaller model would produce the first words ~100 ms sooner. On an Apple M-series or NVIDIA machine the same architecture gets there with native Metal/CUDA paths and far less effort.

---

Setup

Prerequisites: Windows 11 or Linux with an Intel Core Ultra (for the NPU/GPU paths; CPU fallbacks exist but are slow), uv, Ollama with ollama pull granite4.2:3b.

uv sync                                                  # main environment
uv venv --python 3.12 tts_worker/.venv                   # isolated LuxTTS environment
uv pip install --python tts_worker/.venv -r tts_worker/requirements.txt
uv pip install --python tts_worker/.venv --reinstall-package torch --reinstall-package torchaudio --index-url https://download.pytorch.org/whl/cpu torch torchaudio
uv pip install --python tts_worker/.venv openvino

Model files: Granite STT and LuxTTS download from Hugging Face on first use; Kokoro files go in models/kokoro/ (kokoro-v1.0.onnx, voices-v1.0.bin from the kokoro-onnx releases); the speaker model wespeaker_en_voxceleb_CAM++.onnx goes in models/speaker/ (sherpa-onnx releases); Whisper downloads from the OpenVINO Hugging Face org. First start compiles everything (~5 min), later starts take ~90 s.

Run:

uv run python ui/server.py       # console at http://127.0.0.1:8765
uv run python main.py            # terminal mode
uv run python main.py --wav samples/test_utterance_16k.wav --no-audio-out   # offline check

First conversation: Record my voice… (read the script, aim for "good take", Use this voice), enable Voice lock, unmute (Space), talk.

---

Hosting

• Vercel / static hosts can serve only the page (web/). The models need a real machine: run ui/server.py there, expose it with cloudflared tunnel --url http://127.0.0.1:8765, enter that URL in the page's backend field and tick Use THIS browser's microphone. Put Cloudflare Access in front: it speaks in your voice.
• CPU-only VPS (2 vCPU / 8 GB): 10–30 s per turn with this stack; a trimmed variant (whisper-tiny, 0.5B LLM, Kokoro, no cloning) reaches 3–6 s. Not recommended.
• Mac mini M4 24 GB: a good fit (unified memory, Metal, device='mps' in LuxTTS); expect ~600–800 ms to the answer after porting the Intel-specific STT/TTS backends.
• NVIDIA GPU box: every model has native CUDA paths; sub-500 ms is realistic.

---

Configuration (config.yaml)

---

Repository layout

agent/          core package (pipeline, VAD, STT x2, LLM, tools, prefetch, chunker, TTS client, voice lock, audio I/O, metrics)
tts_worker/     LuxTTS worker process + its isolated environment (server.py, lux_fast.py, requirements.txt)
ui/             FastAPI console server, console page, voice-recording page
web/            static hosted copy of the console (generated)
benchmarks/     stt/, llm/, tts/ — the experiments behind every decision
tests/          end-to-end test of the browser-audio WebSocket path
tools/          build_web.py
docs/           architecture.svg
samples/        synthetic test utterances
voices/         your reference recording (git-ignored)
models/         downloaded weights and compiled caches (git-ignored)
main.py, config.yaml, pyproject.toml

Testing and CI

uv run pytest tests/unit -q      # ~1 min, no GPU/NPU/mic: models are replaced by fakes with known latencies
uv run pytest -m speed -rA       # the pipeline's own overhead must stay < 250 ms on top of the model latencies
uv run ruff check .

GitHub Actions: ci.yml (lint, Ubuntu + Windows test matrix, latency budget, hosted-page freshness), tts-worker.yml (the isolated LuxTTS environment resolves and imports; weekly), benchmark.yml (real end-to-end latency on a self-hosted runner with the hardware, on demand, uploads the per-turn tables and reply audio).

Contributing

See CONTRIBUTING.md (dev setup, test layers, how to add an STT/LLM/TTS backend, "bring numbers" rule for latency PRs), SECURITY.md (responsible use of voice cloning) and the issue templates, including a latency report template to grow the hardware table. Licensed under Apache-2.0.

Models and licenses