Low Latency Audio-to-Audio Web Demo

Step 1 builds the WebSocket foundation for a low-latency voice conversation system.

Run

/user_data/audio_voice_demo/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8010

Open:

http://localhost:8010/

Current Protocol

All runtime audio/control traffic uses one WebSocket endpoint:

ws://localhost:8010/ws/audio

Client to server:

• JSON { "type": "client.ready", "inputSampleRate": 16000 }
• Binary PCM16 little-endian mono audio at 16 kHz
• JSON { "type": "stop" }

Server to client:

• JSON server.ready
• JSON audio.stats
• JSON asr.ready, asr.final, asr.ignored
• JSON llm.ready, llm.start, llm.delta, llm.say, llm.end
• JSON tts.ready, tts.start, tts.end
• Binary PCM16 little-endian mono audio at 48 kHz
• JSON demo TTS start/end events

Step 1 Scope

Implemented:

• FastAPI app with /ws/audio
• Browser microphone capture through Web Audio API
• Client-side downsampling to 16 kHz PCM16
• Async receiver, processor, and sender tasks
• Queue-based audio handoff
• Server-to-client PCM16 playback path
• Proxy-aware static assets and WebSocket URL construction

Step 2 VAD

Implemented:

• Independent VAD worker consuming incoming audio chunks from input_audio
• Pluggable VAD engine
• Silero VAD integration when silero_vad and torch are installed
• Energy-based fallback VAD when Silero dependencies are unavailable
• Speech start/end events over WebSocket
• Streaming speech event handoff to asr_events

Optional VAD dependencies:

python3 -m pip install -r requirements-vad.txt

For a CUDA GPU environment, install the matching PyTorch build first, then install requirements-vad.txt.

VAD endpoint tuning:

export VAD_END_SILENCE_SECONDS=0.9
export MIN_UTTERANCE_SECONDS=0.45
export ASR_MIN_RMS=0.006
export ASR_MIN_VOICED_RATIO=0.08

Increase VAD_END_SILENCE_SECONDS if normal pauses split one sentence too early. Increase ASR_MIN_RMS or ASR_MIN_VOICED_RATIO if silence/noise still reaches ASR and causes hallucinated text.

Step 3 ASR Final Utterance Recognition

Implemented:

• Independent ASR worker consuming AsrEvent items from VAD
• Speech-start, speech-chunk, and speech-end handoff from VAD to ASR
• Final utterance transcription on VAD speech end
• LLM handoff only after the complete utterance is recognized
• Optional faster-whisper/CTranslate2 backend
• Placeholder backend when faster-whisper or model path is unavailable

Install ASR runtime dependencies:

python3 -m pip install -r requirements-asr.txt

Breeze ASR 25 must be converted to CTranslate2 format outside this app, then exposed at runtime:

export BREEZE_CTRANSLATE2_MODEL=/path/to/breeze-asr-25-ctranslate2
export ASR_DEVICE=cuda
export ASR_COMPUTE_TYPE=float16
export ASR_LANGUAGE=zh
python3 -m uvicorn app.main:app --host 0.0.0.0 --port 8010

Local demo install status:

• Project virtualenv: .venv
• Breeze ASR 25 CTranslate2 model: models/breeze-asr-25-ct2
• VoxCPM2 model: models/voxcpm2
• Current runtime on this machine: CUDA on RTX 4090
• PyTorch: CUDA 12.8 build

Not implemented yet:

• Barge-in queue clearing

Step 4 TTS

Implemented:

• VoxCPM2 package installed in .venv
• Local openbmb/VoxCPM2 weights downloaded to models/voxcpm2
• Optional OpenFormosa BlueMagpie-TTS backend with the same streaming WebSocket playback path
• Agent UI TTS profile switcher for VoxCPM2 and BlueMagpie TTS
• TTS backend preloaded on FastAPI startup
• Streaming TTS worker consuming tts_text
• VoxCPM2 generate_streaming with inference_timesteps=6
• PCM16 48 kHz audio chunks pushed over the same WebSocket
• demo.tts WebSocket control and front-end Test TTS button

Current CUDA smoke test:

• tts.ready reports voxcpm2
• demo.tts receives tts.start
• Short demo.tts first audio chunk arrives in about 0.48s

For the intended low-latency demo, run with:

export ASR_DEVICE=cuda
export ASR_COMPUTE_TYPE=float16
export TTS_DEVICE=cuda
export TTS_INFERENCE_TIMESTEPS=6
export TTS_SEED=20260612
/user_data/audio_voice_demo/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8010

To keep VoxCPM2 voice identity more stable, the backend fixes the TTS random seed by default. For a stronger fixed voice, provide a reference clip:

export TTS_REFERENCE_WAV=/path/to/reference.wav

Optional BlueMagpie install:

python3 -m pip install -r requirements-bluemagpie.txt
export BLUEMAGPIE_MODEL=/user_data/audio_voice_demo/models/bluemagpie-tts
export BLUEMAGPIE_ALLOW_DOWNLOAD=1
export BLUEMAGPIE_VOICE_MODE=centroid
export BLUEMAGPIE_REFERENCE_WAV=/user_data/audio_voice_demo/models/bluemagpie-ref-8s.wav
export BLUEMAGPIE_INFERENCE_TIMESTEPS=10
export BLUEMAGPIE_CFG_VALUE=2.8
export BLUEMAGPIE_MAX_AUDIO_SECONDS=8.0

Use TTS_PROFILE=bluemagpie to make BlueMagpie the startup default, or switch it from the /agent status panel. The default BLUEMAGPIE_VOICE_MODE=centroid uses the bundled hung_yi_lee speaker centroid because it matches the upstream usage guide more closely than reference-audio cloning.

Or use continuation-style prompting:

export TTS_PROMPT_WAV=/path/to/prompt.wav
export TTS_PROMPT_TEXT="The exact text spoken in the prompt audio."

Not implemented yet:

• Barge-in queue clearing

Step 5 Local LLM Streaming

Implemented:

• Local OpenAI-compatible LLM backend for llama.cpp server
• Streaming /v1/chat/completions SSE parser
• Markup parser for <SAY>...</SAY> and <TOOL_CALL>...</TOOL_CALL>
• Punctuation chunking from streamed <SAY> content into the TTS queue
• Short mock <TOOL_RESULT> events for local testing
• Same-model follow-up after tool results, with the prior spoken acknowledgement preserved as context
• Front-end llm.delta event logging to verify real token streaming
• Google AI Studio API key flow removed from the test UI

Default local LLM endpoint:

http://127.0.0.1:8011/v1/chat/completions

Recommended Gemma 4 test target:

llama-server \
  -m /user_data/audio_voice_demo/models/llm/gemma-4-e4b-it-Q6_K.gguf \
  --host 127.0.0.1 \
  --port 8011 \
  -ngl 99 \
  -c 4096 \
  --flash-attn

FastAPI reads these environment variables:

export LOCAL_LLM_BASE_URL=http://127.0.0.1:8011/v1
export LOCAL_LLM_MODEL=gemma-4-e4b-it
export LOCAL_LLM_TEMPERATURE=0.45
export LOCAL_LLM_MAX_TOKENS=384

If llama.cpp is not running, the backend emits a short spoken fallback so the rest of the ASR-to-TTS pipeline remains testable.