Realtime Voice Demo

Browser voice-chat UI for the huggingface/speech-to-speech backend, speaking the OpenAI Realtime GA protocol over WebSocket (default) or WebRTC (Settings → Transport, env-pinned deploys only — see WebRTC transport).

Both choices run one RealtimeSession adapter over the pinned official @openai/agents package's stock transport classes. The adapter keeps demo-only queue, audio, visualization, device, camera, and metering behavior out of the protocol implementation.

Quick start (local)

1. Start the speech-to-speech backend (from the repo root; see the backend README for more model combinations):

   uv run speech-to-speech serve \
     --stt parakeet-tdt \
     --llm_backend transformers \
     --tts kokoro \
     --model_name "Qwen/Qwen3-4B-Instruct-2507" \
     --llm_device mps \
     --llm_torch_dtype float16 \
     --enable_live_transcription
   

   The realtime server listens on ws://localhost:8765/v1/realtime by default (--host / --port to change).

2. Install the pinned browser SDK and start this app, pointing it at the backend with SPEECH_TO_SPEECH_URL:

   npm ci
   uv pip install -r requirements.txt
   export SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime
   export SERPER_API_KEY=...   # optional; web search is disabled without it
   export STARTUP_GREETING=... # optional; empty disables the automatic greeting
   uv run uvicorn server:app --reload --port 7860
   

   Or with Docker:

   docker build -t s2s-demo .
   docker run -p 7860:7860 -e SPEECH_TO_SPEECH_URL=ws://host.docker.internal:8765/v1/realtime s2s-demo
   

   Docker + host backend: WebSocket and WebRTC need different hostnames. The two transports dial the backend from different network namespaces:

   • WebRTC is dialed server-side — the browser POSTs its SDP offer to the demo's /api/calls proxy, which forwards it from inside the container. There host.docker.internal resolves to your host, so the command above works.
   • WebSocket is dialed client-side — the demo hands the URL straight to the browser, which opens the socket itself. The browser runs on your host, where host.docker.internal is not a real DNS name, so the connection never reaches the backend and the server logs nothing.

   A single Docker SPEECH_TO_SPEECH_URL can therefore only make one transport work at a time (localhost:8765 for WebSocket, host.docker.internal:8765 for WebRTC). To exercise both without swapping the env, run the demo without Docker (the uvicorn command above) so host and container namespaces collapse — then ws://localhost:8765/v1/realtime works for both.

3. Open http://localhost:7860/, click the orb, allow the mic, talk.

Browsers require HTTPS or localhost for getUserMedia() (mic + camera). 127.0.0.1 and localhost both work; plain http://192.168.x.y does NOT.

Smoke-test the backend from the shell:

websocat ws://localhost:8765/v1/realtime
# -> you should get a session.created event back immediately

How it works

1. The adapter creates an official Agents SDK RealtimeSession with the stock WebSocket transport on the configured /v1/realtime URL.
2. The SDK performs session.update using the OpenAI Realtime GA schema.
3. The SDK streams mic audio as PCM16 24 kHz mono base64 chunks (input_audio_buffer.append, one frame every ~40 ms).
4. The browser keeps a bounded, in-memory copy of those sent frames and uses the server VAD boundaries to add replayable user recordings to conversation history. No recording is uploaded or persisted separately.
5. Server pushes response.output_audio.delta (PCM16 24 kHz mono base64) and transcript deltas.

The backend exposes one concurrent session per pipeline unit (--num_pipelines to serve more).

WebRTC transport

With SPEECH_TO_SPEECH_URL set, Settings → Transport offers WebRTC as an alternative to the WebSocket. Same conversation, different plumbing:

1. The SDK's stock WebRTC transport adds the browser mic track and its data channel to an RTCPeerConnection and POSTs the SDP offer to the same-origin /api/calls proxy, which forwards it to the backend's POST /v1/realtime/calls (the OpenAI GA handshake). The proxy exists because the s2s server has no CORS middleware — and it forwards only to the env-pinned URL, never to a client-supplied one, so it can't be used as an open proxy. That's why the toggle is locked to WebSocket when the URL isn't pinned (user-typed URLs, LB mode).
2. Only the handshake goes through the proxy: the negotiated audio (Opus RTP both ways) and the data channel flow directly browser ↔ backend.
3. JSON events on the data channel are the same GA protocol as the WebSocket, minus the audio: mic audio rides the media track (never input_audio_buffer.append, which the backend rejects over WebRTC), and the assistant's voice arrives as a remote audio track (never response.output_audio.delta). Barge-in flushing is server-side.

Backend requirement: the webrtc extra (pip install "speech-to-speech[webrtc]"), otherwise /v1/realtime/calls answers 501 and the handshake fails with a clear message.

Caveats vs. WebSocket:

• User recording replay: conversation-history recordings currently use the exact PCM frames sent through input_audio_buffer.append, so they are available only on the WebSocket transport.
• NAT: host ICE candidates only by default — fine when browser and backend are on the same machine/LAN. Across the internet, set RTC_ICE_SERVERS on this app (a JSON list of RTCIceServer dicts, or comma-separated STUN/TURN URLs; served to the browser via /api/config) and SPEECH_TO_SPEECH_ICE_SERVERS on the backend. There is no TURN relay fallback, so symmetric-NAT setups may still not connect.
• Noise gate: implemented in the WebSocket capture worklet, so it's hidden on WebRTC — the raw mic track (with the browser's own noiseSuppression) is sent instead.
• Camera snapshots are re-encoded to fit one data-channel message (~60 KB), so the model may see a smaller frame than over WebSocket.
• Load-balancer mode is WebSocket-only for now.

Connecting to a backend

Three modes, picked by env (/api/config tells the client which one is active):

• SPEECH_TO_SPEECH_URL env — the mode you want for local use, and the highest priority. The browser connects directly to this realtime WebSocket URL; it's shown read-only in Settings. Setting it disables the load-balancer logic entirely (no /api/session proxy, no queue, no metering, no sign-in). Unlike the LB address it is not a secret. Accepts a full ws(s)://host/v1/realtime URL or a bare host like localhost:8765 (the app adds /v1/realtime).
• Neither env set — Settings → Speech-to-speech server URL: paste a full connect URL or a bare host, and the browser connects to it directly.
• LOAD_BALANCER_URL env — multi-compute deployments only: the browser POSTs the same-origin /api/session proxy, the server forwards to the LB, and the browser dials the per-session compute URL the LB hands back. The LB address never reaches the browser; the Settings URL field is hidden. On OAuth-enabled Spaces, the proxy forwards the signed-in user's HF access token to the LB through X-Reachy-Mini-Authorization so the backend can attribute usage. The token stays server-side; anonymous requests include no credential. Set REQUIRE_LOGIN=true to require Hugging Face sign-in before a visitor can request a session or join the queue.

Settings → Restart reconnects with the current voice, instructions and URL.

Startup greeting

By default, each new connection creates one hidden user item asking the model for a brief greeting, then requests a response. Besides opening the conversation naturally, this warms the same prompt prefix used by the first spoken turn.

Set STARTUP_GREETING to customize the hidden prompt, or set it to an empty value to disable automatic generation. The adapter sends it once through the new RealtimeSession after connection.

Tools

The assistant can call two tools mid-conversation (toggle them from the Tools button, top-right):

• Web search — Google results via Serper.dev, proxied server-side so the key never reaches the browser. Set SERPER_API_KEY as an env var / Space secret. Without it, the tool is disabled unless the user pastes their own key in the Tools panel.
• Camera — while enabled, a live self-view shows bottom-left; when the model calls the tool, the current frame is sent to the vision-language model so it can see what you're showing it.

Usage limits (deployed Space only)

Conversation time is metered per UTC day by sign-in tier (see limiter.py / auth.py), but only on the deployed Space — metering turns on only when BOTH LOAD_BALANCER_URL and SPACE_ID (injected automatically by the HF Space runtime) are present. Running locally — even with LOAD_BALANCER_URL exported — leaves the app unmetered. Tunable via env:

PRO members are always unlimited. Members of cerebras, HuggingFaceM4, smolagents, and pollen-robotics are unlimited out of the box (shown as "Team", not "PRO"); set UNLIMITED_ORGS=my-team to add more. Matched case-insensitively against the user's organisations from HF OAuth.

Settings (stored in localStorage)

LocalStorage keys are namespaced s2s.ws.* (plus s2s.transport for the transport pick, and s2s.audio.inputId / s2s.audio.outputId for devices).

Files

Audio pipeline notes

• Input: getUserMedia({ echoCancellation, noiseSuppression, autoGainControl }) feeds the mic-capture worklet at the AudioContext rate. The worklet resamples to 24 kHz (boxcar lowpass + decimation on the 48 -> 24 fast path, linear interpolation fallback for odd rates) and packs Int16 LE.
• User replay: the WebSocket client retains only a bounded copy of PCM it actually sends. speech_started / speech_stopped timestamps select each utterance, which is wrapped as an in-memory WAV and attached to the user row. Starting playback temporarily mutes outgoing mic audio to prevent feedback.
• Output: response.output_audio.delta decodes to Int16 -> Float32 and is posted to the audio-playback worklet. The worklet maintains a per-context ring buffer, linearly interpolates 24 -> 48, and applies short 32-frame fades on entry/exit to suppress clicks.
• Barge-in: when the server VAD detects user speech mid-response (input_audio_buffer.speech_started while ai-speaking), the client posts { kind: "clear" } to the playback worklet to wipe the queue immediately. The server itself cancels the in-flight response.

Credits

• Backend: huggingface/speech-to-speech
• UI verbatim from amir-tfrere/minimal-conversation-app-s2s-backend (Pollen Robotics × Hugging Face)