Voice Light

Voice Light is an end-to-end research project for building a natural, low-latency streaming voice agent: generate and label conversational data, train language and turn-taking adapters, evaluate their behavior, integrate the useful pieces into a full-duplex cascade, and deploy the result on scale-to-zero GPU compute.

Try the live demo · Read the paper · Read the deployment record · Explore the models and datasets

The browser streams microphone audio directly to a provider-neutral compute service on Modal. Nemotron performs streaming ASR and supplies the encoder features used by a small causal turn-taking adapter. Qwen produces conversational responses and structured tool calls, Kyutai streams speech, and browser playback acknowledgements ensure that only audio the user actually heard enters durable conversation history.

What this project covers

Voice Light treats the deployed agent as the last stage of a reproducible research pipeline rather than an isolated demo.

1. Synthetic spoken tool-use data. A pinned Qwen3.6-27B teacher generated 3,999 typed conversations across eight behavior families and five speech styles. The generator models ordinary dialogue, search, calculation, time lookup, failures, corrections, and sequential tool rounds while keeping audible speech separate from structured protocol data.
2. Tool-use language-model fine-tuning. A rank-16 LoRA trained Qwen3-1.7B to say a short spoken bridge, emit a structured call, consume its result, continue naturally, and avoid unnecessary tools. The adapter and a merged BF16 checkpoint are published independently of the runtime.
3. Synthetic turn-taking audio. Typed conversation plans describe completion, HOLD pauses, backchannels, responses, and interruptions. User speech is synthesized and measured before the pipeline compiles causal 20-second views with dense 80 ms labels; virtual assistant speech is a timeline signal, not leaked audio.
4. Automatic annotation of human conversations. The ingestion pipeline registers and hashes two-speaker recordings, runs independent ASR and VAD analysis, aligns channels, derives conversation regions and quality evidence, and preserves supplied human annotations as a separate source of evidence. Conversation-disjoint splits prevent windows from the same call crossing train, validation, and test.
5. Causal turn-taking adapter training. A roughly 183k-parameter GRU adapter consumes taps from the frozen streaming Nemotron encoder plus assistant-playback state. Training starts on synthetic interaction semantics, then fine-tunes on human conversation with synthetic replay; human validation, not synthetic fit, selects the checkpoint.
6. Evaluation and integration. Offline benchmarks lock thresholds before test evaluation and report false cutoffs, recall, calibration, and latency. The deployed hybrid controller combines Silero onset, reversible ducking, learned floor-taking evidence, backchannel resume, and a conservative fallback instead of treating every sound as an irreversible interruption.

The detailed records are in synthetic tool-use generation, LoRA training, tool-use evaluation, synthetic turn-taking construction, human corpus preparation, turn-taking training, and the locked turn-detection benchmark.

Runtime architecture

Browser microphone
    │ 16 kHz PCM + playback acknowledgements
    ▼
Modal WebSocket (/v1/voice)
    ├── GPU 0: streaming Nemotron ASR ── shared encoder taps ── turn adapter
    │          Qwen3 4B conversation model + Qwen3 0.6B search summarizer
    ├── GPU 1: Kyutai streaming TTS
    └── typed orchestration: prediction, tools, cancellation, playback barriers
                         │
                         ▼
Browser audio queue + live interaction diagnostics

The production endpoint admits one session at a time and scales to zero after two idle minutes. Model caches live in a persistent Modal volume; model processes are persistent inside a warm container, but an idle deployment still pays the CUDA and model-initialization cost on its next cold start. The static frontend is hosted separately on GitHub Pages.

The deployed conversation model is the stronger pinned Qwen/Qwen3-4B-Instruct-2507, not the 1.7B experimental LoRA. This preserves general conversational quality while keeping the LoRA as a reproducible result and comparison point. The same principle applies to endpointing: the learned adapter contributes interaction evidence, but current production latency still relies heavily on Silero and speculative generation because the locked learned completion policy did not beat the stronger timing baselines.

Measured results

Tool-use fine-tuning

On the original balanced 80-record holdout, the final 1.7B proof-of-concept adapter improved:

These are held-out results from a synthetic experiment, not evidence of general factual accuracy. A later 240-state checkpoint comparison selected an earlier epoch and also exposed remaining tool selection errors; the complete lineage and both evaluations are documented in the results report and published model card.

Turn detection

The locked real-conversation test contained 1,673 candidates. At its selected policy, the Voice Light checkpoint produced one false cutoff among 37 HOLD cases, but crossed its learned threshold for only 205 of 1,636 EOT cases. Most endpoints therefore used the timeout, and the adapter did not outperform the Silero or LiveKit timing policies on this protocol. The small HOLD denominator and known benchmark provenance constraints are reported with the results; no superiority claim is made from synthetic interaction validation.

Public voice demo

Across three unscripted operator-run microphone sessions, 36 response turns were measured. The first server audio packet arrived within 800 ms of the final VAD endpoint for 21 turns. One session exported complete component telemetry for 13 turns.

This is an instrumented case study run by one operator, not a controlled user evaluation or a population latency distribution. The paper and final deployment record contain the measurement definitions, limitations, and development smoke-test history.

Published artifacts

Other human sources retain their original access, license, privacy, and redistribution constraints. Raw source mappings, credentials, private annotations, and local training artifacts are excluded from Git. See corpus preparation before using any human data.

Documentation

Data and training

• Synthetic tool-use generation
• Teacher-led generation and reproducibility
• Tool-use LoRA training
• Synthetic conversational turn-taking dataset
• Human training corpus preparation and validation
• Turn-taking adapter training
• Full-recording ASR and automatic annotation

Evaluation and design

• Tool-use fine-tuning results
• Turn-detection benchmark
• Completion retraining
• Latency and endpoint baselines
• Natural interaction design study

Runtime and deployment

• Streaming voice-agent architecture
• Provider-neutral compute service
• Modal deployment runbook and test record
• Historical Modal restoration design
• Vast.ai compute deployment

Running the project

For the local data and analysis application, start Postgres, apply migrations, and run the FastAPI server as described in the compute and local-app guide. Open http://127.0.0.1:8000 for the dataset tools or /voice-agent for the local voice client.

For the production voice stack, follow the Modal runbook. It covers the pinned models, persistent cache volume, adapter checkpoint packaging, GPU topology, secrets, deployment command, scale-from-zero smoke test, and browser verification. Credentials are supplied through Modal secrets and are never committed.

Known limitations

• Scale-from-zero readiness is approximately 32–55 seconds in the final topology; cached weights do not eliminate Python, CUDA, and model initialization.
• The learned completion head has low recall at its conservative locked threshold, so production endpointing remains a hybrid policy rather than an adapter-only result.
• Tool use and factual answers remain model fallible. Tool schemas and results are validated, but the language model can still select the wrong tool or misstate a result.
• Tool bridges and post-result continuations are separate semantic speech segments and can retain a small audible seam.
• The public demo is single-session and English-first. Microphone, network, browser scheduling, and Modal host variation all affect perceived latency.

Paper

Voice-Light: A Full-Duplex Cascaded Voice Agent with Causal Turn-Taking and Speculative Generation connects the data, training, evaluation, integration, deployment, and microphone case study in one narrative. The corresponding GitHub release contains the PDF and exact arXiv source package; the maintained LaTeX source and build instructions are in docs/technical-report.

@article{braun2026voicelight,
  title = {Voice-Light: A Full-Duplex Cascaded Voice Agent with Causal Turn-Taking and Speculative Generation},
  author = {Braun, Bertil},
  journal = {arXiv preprint arXiv:2609.20995},
  year = {2026},
  doi = {10.48550/arXiv.2609.20995},
  url = {https://arxiv.org/abs/2609.20995},
}