open-realtime-translate

A fully local, real-time speech interpretation server for MLX, CUDA, and CPU devices, wire-compatible with the OpenAI Realtime Translation API (/v1/realtime/translations).

open-realtime-translate turns live speech into translated speech and transcripts on the user's machine. A Realtime client can point to the local server in place of a hosted API endpoint: audio is received, transcribed, translated, and synthesized on-device, while API keys and hosted audio services stay outside the runtime path. Apple Silicon uses the MLX profile; NVIDIA systems can use CUDA; CPU mode provides a smaller, portable baseline.

Core capabilities

Supported languages

The system composes language coverage across three stages. Source speech is recognized by the active ASR backend, translated by the active MT backend, and rendered by the active TTS backend. Source language is detected per audio span, with an optional session-level ASR hint at session.audio.input.transcription.language; target language defaults to en and is changed per session with session.audio.output.language within the active backend's range.

Usable speech-to-speech pairs are the intersection of these stage-level constraints; the exact tables live in ort/languages.py. Multilingual translation is handled by the default translategemma backend.

Input languages (Qwen3-ASR, auto-detected)

Output languages (depends on the TTS backend)

End-to-end checks have covered the default Qwen3-TTS path on zh→en, en→es, and zh→ja. The optional Kokoro path has also been checked across zh→ja, en→es, es→fr, en→zh, zh→hi, and ja→en.

Performance

The following measurements were taken on an Apple M5 Pro with model files already cached. They describe a verified MLX reference profile and are intended as local reference points, not service-level guarantees. CUDA and CPU throughput depends strongly on the selected ASR model, GPU memory, Torch dtype, and TTS backend.

MLX reference measurement (Qwen3-ASR-0.6B → TranslateGemma-4b-it → Qwen3-TTS-12Hz-1.7B-Base):

TTS backends compared (cold start, resident memory, real-time factor; lower RTF is faster):

Kokoro remains available when the priority is minimum latency or a smaller resident footprint. Qwen3-TTS is the default because it gives the strongest voice identity behavior and carries the translation across languages in the reference speaker's timbre.

Accuracy is strongest for conversational speech whose meaning unfolds in sequence. Very long single sentences with cross-clause reordering can fragment in incremental mode. For maximum accuracy, use whole-utterance mode (ORT_INCREMENTAL=0), which exchanges latency for complete-sentence context.

How it works

flowchart LR
    client["Realtime client"]
    mic["Live speech"]
    ws["WebSocket<br/>base64 PCM16"]
    rtc["WebRTC<br/>Opus audio track"]
    decode["Decode + resample<br/>24 kHz wire -> 16 kHz internal"]
    vad["VAD + silence tracking"]
    commit{"Commit span?"}
    queue["Ordered span queue"]
    runtime{"Runtime profile<br/>auto / mlx / cuda / cpu"}
    mlx["MLX profile<br/>Qwen3-ASR + TranslateGemma MLX + Qwen3-TTS MLX"]
    cuda["CUDA profile<br/>Qwen3-ASR Torch + TranslateGemma Torch + Qwen3-TTS Torch"]
    cpu["CPU profile<br/>Whisper CPU + TranslateGemma Torch + Supertonic-3"]
    asr["ASR<br/>source speech -> source text"]
    context["Committed context<br/>source -> target history"]
    mt["Translate<br/>source text -> target text"]
    ref["Rolling voice reference<br/>latest ASR audio + transcript<br/>0-10 seconds"]
    pad["TTS handoff padding<br/>pad copy to 3 seconds if needed"]
    tts["TTS / voice clone<br/>target text -> target speech"]
    audio["Translated speech<br/>24 kHz PCM16 / Opus"]
    text["Transcript events<br/>source + target deltas"]

    mic --> ws
    mic --> rtc
    ws --> decode
    rtc --> decode
    decode --> vad
    vad --> commit
    commit -- "short pause or max span" --> queue
    commit -- "keep listening" --> vad
    runtime -- "Apple Silicon / ORT_RUNTIME=mlx" --> mlx
    runtime -- "ORT_RUNTIME=cuda" --> cuda
    runtime -- "ORT_RUNTIME=cpu" --> cpu
    mlx -.-> asr
    mlx -.-> mt
    mlx -.-> tts
    cuda -.-> asr
    cuda -.-> mt
    cuda -.-> tts
    cpu -.-> asr
    cpu -.-> mt
    cpu -.-> tts
    queue --> asr
    asr --> ref
    asr --> context
    context --> mt
    asr --> text
    mt --> text
    mt --> tts
    ref --> pad
    pad --> tts
    tts --> audio
    audio --> client
    text --> client

The processing path is deliberately staged. Audio capture and network I/O can continue while inference runs. MLX model initialization and inference are serialized on a dedicated Apple Silicon thread; CUDA and CPU backends use their native PyTorch, CTranslate2, or ONNX execution paths. The incremental committer finalizes short spans at clause-boundary pauses, which lets the server speak before a long source sentence has fully ended.

Model stack

All profiles keep the same VAD, segmentation, OpenAI-compatible events, and context-aware translation loop. Individual stages remain replaceable with ORT_ASR_BACKEND, ORT_TRANSLATOR_BACKEND, and ORT_TTS_BACKEND; mock backends are reserved for protocol tests.

Transport

Wire-contract details are documented in PROTOCOL.md.

Voice cloning

The qwen3_tts backend uses Qwen3-TTS-12Hz-1.7B-Base to synthesize each translation in the current source speaker's voice. Each session starts with an empty reference buffer. After ASR succeeds for a committed span, the server appends that span's source audio and ASR transcript to a rolling voice reference. The buffer itself stores only real ASR audio and is capped at the newest 10 s. On each TTS call, the backend makes a TTS-ready copy of the latest reference; only that copy is right-padded to 3 s when the real reference is shorter. The output language may differ from the reference language. This backend is the default on MLX and CUDA; CPU mode defaults to Supertonic-3 because it is smaller and does not require voice cloning.

# Optional: provide a static reference for direct backend calls or warmup
ORT_TTS_BACKEND=qwen3_tts \
ORT_QWEN3_TTS_REF_AUDIO=/path/to/voice.wav \
ORT_QWEN3_TTS_REF_TEXT="exact transcript of the reference clip" \
  uv run --extra mlx python -m ort.app --warmup

Use --extra nonmlx for the CUDA implementation. In normal translation sessions, explicit reference files are no longer required because the source speaker reference comes from the ASR input itself. A clean microphone signal and accurate ASR transcript improve the clone quality.

Requirements

Quick start

# 1. Install dependencies (creates .venv, Python 3.12)
# Apple Silicon / MLX:
uv sync --extra dev --extra mlx

# 2. Start the server (--warmup preloads models before the first connection)
uv run --extra mlx python -m ort.app --host 127.0.0.1 --port 8000 --warmup

# 3. Translate a Chinese sentence into English speech in out.wav
uv run --extra mlx python examples/client.py --text "今天天氣很好，謝謝你。" --out out.wav

Point an OpenAI Realtime client at:

ws://127.0.0.1:8000/v1/realtime/translations

For CUDA or CPU, install the non-MLX profile instead:

# CUDA: Qwen3-ASR Torch + TranslateGemma Torch + Qwen3-TTS Torch
uv sync --extra dev --extra nonmlx
ORT_RUNTIME=cuda uv run --extra nonmlx python -m ort.app --host 127.0.0.1 --port 8000 --warmup

# CPU: Whisper CPU + TranslateGemma Torch + Supertonic-3
uv sync --extra dev --extra nonmlx
ORT_RUNTIME=cpu uv run --extra nonmlx python -m ort.app --host 127.0.0.1 --port 8000 --warmup

The mlx and nonmlx extras are intentionally mutually exclusive. The MLX stack and the non-MLX Qwen packages currently resolve to different Transformers / Hugging Face dependency ranges, so each runtime profile should have its own environment.

Browser demo

A Gradio UI for recording from the microphone or uploading a clip, viewing the source and translated transcripts, and playing back the translated speech:

uv sync --extra demo --extra mlx           # one-time: install Gradio for the MLX profile
uv run --extra mlx python -m ort.app --warmup &        # start the server
uv run --extra demo python examples/gradio_demo.py      # open the printed local URL

Use --extra nonmlx in place of --extra mlx when the demo should run against CUDA or CPU.

Configuration

Use lighter ASR/TTS around the default translator:

ORT_ASR_BACKEND=whisper ORT_TTS_BACKEND=say \
  uv run --extra mlx python -m ort.app --warmup

Testing

Backend benchmarks live in scripts/, including bench_comprehensive.py for the full-stack and TTS comparison, and verify_qwen3_tts.py for voice-clone fidelity.

Model licenses

Third-party model weights remain governed by their publishers' terms. Review those terms before redistribution, hosted use, or commercial deployment. License metadata below was checked against the linked public model pages on 2026-05-19.

Permissive licenses such as Apache-2.0 and MIT still require preserving notices. Gemma and OpenRAIL-M terms include additional use and redistribution conditions.