Prodigy

A high-performance, real-time speech-to-speech system designed for low-latency telephony communication. Prodigy integrates Whisper (ASR), LLaMA (LLM), and a generic TTS stage (with hot-pluggable Kokoro, NeuTTS, VITS2, or Matcha-TTS engines) into a linear microservice pipeline, using a standalone SIP client as an RTP gateway. Optimized for Apple Silicon (CoreML/Metal) with no PyTorch runtime dependency.

An optional RAG sidecar (tomedo-crawl) connects to a Tomedo electronic medical records (EMR) server, crawls patient data, and feeds LLaMA with per-caller context so the AI can greet patients by name and give medically-informed responses.

Architecture

                         Telephony Network
                              |
                         [SIP Client] ──────────────────────────► tomedo-crawl
                          /        \           POST /caller           (port 13181)
                    RTP in          RTP out                              │
                        |              ^                    GET /caller  │
                       IAP            OAP              GET /query        │
                        |              ^                                 ▼
                       VAD            TTS stage                    [Ollama embed]
                        |              ^  ▲                        [Vector store ]
                        |              |  │ (engine dock, port 13143)
                        |              |  └─ Kokoro / NeuTTS / VITS2 / Matcha engine
                     Whisper ────► LLaMA ◄──────────────────────────────┘
                                                    RAG context injection

                     [Frontend] (web UI + log aggregation)

The pipeline is a linear chain of C++ programs. Every adjacent pair communicates over two persistent TCP connections (management + data). The frontend manages all services and provides a web UI at http://0.0.0.0:8080/.

tomedo-crawl is a sidecar — it is not in the audio data path. Communication with other pipeline services is via its own HTTP REST API (port 13181).

Requirements

• OS: macOS Apple Silicon (M1/M2/M3/M4)
• Language: C++17, Python 3.9+
• Build: CMake 3.22+, Ninja (recommended)
• Dependencies (installed automatically by runmetoinstalleverythingfirst):
  • whisper.cpp (compiled with CoreML + Metal)
  • llama.cpp (compiled with Metal)
  • espeak-ng (brew install espeak-ng)
  • macOS frameworks: Accelerate, Metal, CoreML, Foundation

Quick Install & Build

# Step 1: Install everything (Homebrew, Miniconda, models, CoreML exports)
./runmetoinstalleverythingfirst

# Step 2: Build all services
./runmetobuildeverything

# Step 3: Launch
cd bin && ./frontend
# Web UI: http://localhost:8080

runmetobuildeverything auto-clones whisper-cpp and llama-cpp if missing, detects Ninja for fast parallel builds, and bypasses the macOS Xcode license check using the Command Line Tools SDK directly.

Manual Build

# Build whisper.cpp (CoreML + Metal, static)
cmake -G Ninja -S whisper-cpp -B whisper-cpp/build \
  -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF \
  -DWHISPER_COREML=ON -DGGML_METAL=ON \
  -DWHISPER_BUILD_TESTS=OFF -DWHISPER_BUILD_EXAMPLES=OFF
cmake --build whisper-cpp/build -j

# Build llama.cpp (Metal, static)
cmake -G Ninja -S llama-cpp -B llama-cpp/build \
  -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF \
  -DGGML_METAL=ON
cmake --build llama-cpp/build -j

# Build Prodigy
cmake -G Ninja -S . -B build \
  -DCMAKE_BUILD_TYPE=Release -DKOKORO_COREML=ON -DBUILD_TESTS=ON
cmake --build build -j

CMake Options

Models

All model files are placed in bin/models/. runmetoinstalleverythingfirst downloads and prepares all of these automatically.

Whisper (ASR)

LLaMA (LLM)

Kokoro (TTS)

Located in bin/models/kokoro-german/:

NeuTTS (alternative TTS)

Located in bin/models/neutts-nano-german/:

VITS2 / Piper (alternative TTS)

Located in bin/models/vits2-german/:

Download with python3 scripts/setup_vits2_models.py --output-dir bin/models/vits2-german.

Matcha-TTS (alternative TTS)

Located in bin/models/matcha-german/coreml/:

Export with python3 scripts/export_matcha_models.py --checkpoint <path> --output-dir bin/models/matcha-german/coreml.

Moshi & German STT (real-time speech-to-speech)

Located in bin/models/ and /Users/whisper/tokenizer/:

The Moshi backend is built from kyutai-labs/moshi (Rust) with Metal patches for Apple Silicon. The Q8 GGUF model is required on 16GB Macs — BF16 safetensors (15GB) causes swap thrashing.

German STT Fine-Tuning Info (stt-1b-de): The kyutai/stt-1b-en_fr base model was successfully fine-tuned on ~600 hours of German speech on Modal H200 for 3000 steps. The model reached excellent convergence (final train loss ~0.146). The text tokenizer has been extended to 8004 vocabulary entries to natively support German characters (ä, ö, ü, ß).

Performance (Apple M4 Mac mini, 16GB, Q8 GGUF + Metal):

• Steady-state LM step: ~101ms (12.5Hz frame rate = 80ms budget → 1.26× real-time)
• Model warmup: ~4 seconds
• First step: ~2.2s (Metal shader compilation)

Neural G2P (German grapheme-to-phoneme)

Located in bin/models/g2p/:

Export with python3 scripts/export_g2p_model.py --output-dir bin/models/g2p. Used by Kokoro and Matcha when --g2p neural is set; falls back to espeak-ng for non-German input.

Port Map

All services bind to 127.0.0.1:

---

Services

1. SIP Client (bin/sip-client)

RTP gateway and SIP stack. Handles SIP registration with Digest authentication, incoming/outgoing call management, and routes raw RTP audio between the telephony network and the internal pipeline.

Key behaviors:

• Minimal SIP stack over raw UDP (port 5060 by default)
• MD5 Digest authentication with WWW-Authenticate challenge parsing
• Re-registers every 60 seconds
• Multi-line: supports N simultaneous SIP registrations (--lines 0 is valid for test-only mode)
• Inbound RTP forwarded to IAP as raw Packet frames (12-byte RTP header included; IAP strips it)
• Outbound G.711 frames from OAP wrapped in RTP headers (seq, timestamp, SSRC) and sent via UDP
• Stale call auto-hangup after 60 seconds of no RTP traffic
• RTP port base: 10000, incremented by 2 per call

Command-Line Parameters:

Runtime Commands (cmd port 13102):

---

2. Inbound Audio Processor (bin/inbound-audio-processor)

Converts G.711 μ-law telephony audio (8kHz) to float32 PCM (16kHz) for the VAD service.

Signal chain:

1. G.711 μ-law decode: 256-entry ITU-T lookup table; each byte → float32 in [-1.0, 1.0]
2. 8kHz→16kHz upsample: 15-tap Hamming-windowed sinc FIR half-band filter (cutoff ~3.8kHz, ~40dB stopband). Zero-stuffs input, then filters to remove spectral copies above 4kHz.

Each 160-byte RTP payload (20ms @ 8kHz) produces 320 float32 samples (20ms @ 16kHz). Continues processing and discards output if VAD is unavailable; auto-reconnects when VAD comes back online.

Command-Line Parameters:

Runtime Commands (cmd port 13112):

---

3. VAD Service (bin/vad-service)

Energy-based Voice Activity Detection. Segments continuous 16kHz PCM into speech chunks (0.5–8 seconds) for Whisper.

Algorithm:

• Adaptive noise floor: EMA update (alpha=0.05) during silence frames; time constant ~1 second
• Onset detection: requires 3 consecutive frames above threshold × noise_floor to confirm speech start
• End detection: 400ms of consecutive sub-threshold frames triggers speech-end
• Micro-pause detection: short pauses (~400ms) between words trigger early submission rather than waiting for full silence — reduces Whisper inference latency since inference time scales with chunk length
• Smart-split: when max chunk length is reached during speech, finds the lowest-energy frame near the boundary to avoid cutting mid-word
• Pre-speech context: 400ms (8 frames × 50ms) before confirmed onset is prepended to each chunk
• RMS energy gate: chunks with RMS < 0.005 discarded as near-silence
• SPEECH_ACTIVE/SPEECH_IDLE signals: broadcast downstream to the TTS stage (teed to the docked engine) and OAP for TTS interruption and SPEECH_IDLE-driven warm-up

Command-Line Parameters:

Runtime Commands (cmd port 13117):

---

4. Whisper Service (bin/whisper-service)

Automatic Speech Recognition (ASR). Receives pre-segmented speech chunks from VAD and returns transcribed text to LLaMA.

Inference details:

• Backend: whisper.cpp with CoreML ANE (Apple Neural Engine) + Metal fallback
• Decoding: Greedy strategy (not beam search). On 2–8s segments, greedy is 3–5× faster than beam_size=5 with negligible accuracy difference. Temperature fallback with temp_inc=0.2 handles uncertain segments.
• Telephony-optimized parameters: no_speech_thold=0.9 (prevents early decoder stop on G.711-degraded audio), entropy_thold=2.8 (tolerant of codec uncertainty)
• No audio normalization: audio passed directly to Whisper (matches whisper-cli defaults for optimal accuracy on G.711 input)
• RMS energy pre-check: rejects chunks with RMS < 0.005 to prevent hallucinations on near-silence
• Packet buffering: if LLaMA is disconnected, buffers up to 64 transcription packets and drains them on reconnect
• Hallucination filter (default OFF, runtime-toggleable): exact-match detection of common Whisper hallucination strings (e.g., "Untertitel", "Copyright", "Musik"); repetition detection; trailing suffix stripping

Command-Line Parameters:

Runtime Commands (cmd port 13122):

---

5. LLaMA Service (bin/llama-service)

Generates a spoken German reply from transcribed text using Llama-3.2-1B-Instruct.

Inference details:

• Model: Llama-3.2-1B-Instruct Q8_0 GGUF, all layers on Metal GPU (n_gpu_layers=-1)
• Template: llama_chat_apply_template() — uses the model's built-in chat template for correct role tagging; no manual prompt formatting
• Sampling: Greedy (llama_sampler_init_greedy). Max 96 tokens per response. Stops at sentence-ending punctuation (., ?, !) or EOS.
• Context: 2048 tokens, 4 threads
• German system prompt: enforces always-German, max 1 sentence / 15 words, polite and natural tone. ~320ms average latency on Apple M-series.
• Clause-boundary streaming: triggers TTS synthesis at clause boundaries (,, ;, em-dash, en-dash, -) as well as sentence-end punctuation. Reduces perceived latency by ~100–200 ms by starting synthesis before the full sentence is generated. Up to 4 early-streaming chunks per response (MAX_EARLY_STREAM_CHUNKS = 4).
• Session isolation: each call gets its own LlamaCall struct with independent message history and KV cache sequence ID. Context cleared on CALL_END.
• Shut-up mechanism: SPEECH_ACTIVE from VAD aborts active generation immediately (~5–13ms interrupt latency). Worker loop defers new responses while speech is active.
• Tokenizer resilience: retries with progressively larger buffer (up to 4×) if llama_tokenize() returns a negative value

Command-Line Parameters:

Runtime Commands (cmd port 13132):

---

6. TTS Stage (bin/tts-service) and Engines

The TTS stage is a generic pipeline node that sits between LLaMA and OAP. It owns the interconnect sockets (mgmt 13140, data 13141, cmd 13142) and a dedicated engine dock on port 13143. Concrete TTS engines (bin/kokoro-service, bin/neutts-service, bin/vits2-service, bin/matcha-service) are not pipeline nodes — they are client processes that connect to the dock, authenticate via a one-line JSON HELLO, and stream audio back through it.

Engine slot model (last-connect-wins). The dock holds at most one active engine. State transitions:

                 HELLO ok             TCP close / SHUTDOWN ack
  [NO ENGINE] ─────────────▶ [ACTIVE=X] ─────────────────────▶ [NO ENGINE]
                                  │
                                  │  new engine Y completes HELLO ok
                                  ▼
                            [SWAPPING X→Y]
                                  │  dock sends CUSTOM SHUTDOWN to X;
                                  │  X closes TCP (≤ 2 s) or is force-closed
                                  ▼
                              [ACTIVE=Y]

When no engine is docked, LLaMA text frames are dropped with a rate-limited WARN log; mgmt signals (CALL_END / SPEECH_ACTIVE / SPEECH_IDLE) are still auto-forwarded to OAP. On every swap the dock emits a CUSTOM FLUSH_TTS mgmt frame to OAP so residual PCM is discarded before the new engine's audio arrives.

Engine dock protocol (TCP, loopback only):

1. Engine connects to 127.0.0.1:13143.
2. Engine sends one-line JSON HELLO: {"name":"kokoro","sample_rate":24000,"channels":1,"format":"f32le"}\n.
3. Dock replies OK\n (accepts) or ERR <reason>\n (rejects — the current active engine is untouched).
4. After OK, frames are tag-prefixed: 0x01 = serialized data Packet, 0x02 = mgmt (MgmtMsgType + optional payload). The dock ferries LLaMA→engine text, engine→OAP audio, mgmt signals, and PING/PONG keepalives.
5. On receipt of CUSTOM SHUTDOWN the engine process joins its workers, releases model handles, and calls std::_Exit(0).

Runtime Commands (TTS dock cmd port 13142):

6a. Kokoro engine (bin/kokoro-service)

Text-to-speech using the Kokoro model. Receives text from the dock and streams 24kHz float32 PCM audio back through it. No PyTorch dependency — all inference via CoreML on Apple Neural Engine.

Phonemization pipeline:

1. espeak-ng (via libespeak-ng) converts input text → IPA phoneme string. Language auto-detected (de / en-us) via detect_german().
2. Phoneme cache (LRU, 10,000 entries): avoids re-running espeak-ng for repeated phrases.
3. KokoroVocab: greedy longest-match scan (up to 4 chars per token, UTF-8 aware) maps phonemes → int64 token IDs from vocab.json. Input padded to 512 tokens.

Two-stage CoreML inference:

• Stage 1 — Duration model (kokoro_duration.mlmodelc): predicts per-phoneme durations, generates alignment tensors (pred_dur, d, t_en, s, ref_s). Style encoding from <voice>_voice.bin (256-dim reference embedding).
• Stage 1b — F0/N predictor (kokoro_f0n_{3s,5s,10s}.mlmodelc): three bucketed models (3s/5s/10s) predict fundamental frequency (f0_pred) and voicing (n_pred) from the duration model's d and s outputs. Bucket selected by utterance length. These condition the harmonic/noise excitation signal — without them, speech sounds hoarse/unvoiced.
• Stage 2 — Decoder (decoder_variants/*.mlmodelc): split decoder generates the audio waveform from alignment tensors + F0/N conditioning. All models run with MLComputeUnitsAll (ANE + GPU + CPU).

Audio output processing:

• normalize_audio(): scales to 0.90 peak ceiling (skips near-silent audio and already-normalized output)
• apply_fade_in(): 48-sample linear ramp at onset to prevent click artifacts
• Sends audio to OAP in 4800-sample chunks (200ms @ 24kHz) for smooth buffer filling

SPEECH_ACTIVE handling: Abandoned synthesis immediately if VAD signals caller speech. Per-call synthesis threads, so multi-line calls synthesize in parallel.

Prosody state carryover: Each call's ref_s_out (256-dim style tensor) from chunk N is fed as ref_s input to chunk N+1's duration model. This preserves intonation continuity across synthesized chunks without any model re-export.

Command-Line Parameters:

Runtime Commands (Kokoro engine cmd port 13144):

On a CUSTOM SHUTDOWN frame from the dock the engine joins its synthesis workers and exits; it does not restart the pipeline.

---

6b. NeuTTS engine (bin/neutts-service)

Alternative TTS engine using the NeuTTS Nano German model. Like Kokoro it is a dock client — the dock's single engine slot means only one TTS engine serves traffic at a time, and starting a second engine transparently swaps it in (last-connect-wins).

Inference pipeline:

1. espeak-ng converts input text → IPA phonemes (language de, with stress markers)
2. Builds a NeuTTS prompt: user: Convert the text to speech:<|TEXT_PROMPT_START|>{ref_phones} {phones}<|TEXT_PROMPT_END|>\nassistant:<|SPEECH_GENERATION_START|>{ref_codes}
3. Tokenize and feed to NeuTTS backbone (llama.cpp, Q4_0 GGUF) with temperature=1.0, top_k=50 autoregressive sampling
4. Extract <|speech_N|> tokens as integer codec codes
5. Stop at <|SPEECH_GENERATION_END|> or EOS
6. Decode codes through NeuCodec CoreML decoder → 24kHz float32 PCM

Reference voice: Pre-computed codec codes (ref_codes.bin) and phonemized text (ref_text.txt) loaded at startup to define voice timbre and style.

Audio post-processing: Same as Kokoro — normalize to 0.90 peak, 48-sample fade-in.

Command-Line Parameters:

Runtime Commands (NeuTTS engine cmd port 13174):

On a CUSTOM SHUTDOWN frame from the dock the engine joins its synthesis workers and exits.

---

6c. VITS2 engine (bin/vits2-service)

Alternative TTS engine built on Piper TTS via the libpiper C API. Uses ONNX Runtime internally — no CoreML required. Suitable for high-quality German synthesis with fast ISTFT-based vocoding.

Inference pipeline:

1. piper_create() loads a Piper .onnx model + .onnx.json config from $WHISPERTALK_MODELS_DIR/vits2-german/
2. piper_synthesize_start() / piper_synthesize_next() loop produces float32 PCM chunks
3. PCM resampled to 24kHz if model sample rate differs; normalized and chunked to kTTSMaxFrameSamples
4. Audio sent to the dock via EngineClient::send_audio() with 12-byte header per tts-common.h

Phonemization: Piper handles phonemization internally via its bundled espeak-ng. Optional neural G2P pre-phonemization for German when --g2p neural is set.

Command-Line Parameters:

Runtime Commands (VITS2 engine cmd port 13175):

---

6d. Matcha-TTS engine (bin/matcha-service)

Alternative TTS engine based on Matcha-TTS (flow-matching acoustic model) + HiFi-GAN vocoder, both exported to CoreML. The ODE flow is baked into a static CoreML graph (10 Euler steps unrolled at export time) — no iterative solver at runtime.

Inference pipeline (5 stages):

1. Phonemize: espeak-ng or neural G2P → IPA; phoneme cache (LRU)
2. Encoder (matcha_encoder.mlmodelc): text → latent acoustic sequence (512-token fixed input, ANE)
3. Noise sample: Gaussian noise via per-call std::mt19937 (deterministic, seeded per call_id)
4. Baked ODE flow (matcha_flow_{3s,5s,10s}.mlmodelc): noise + latent → mel-spectrogram (bucket selected by utterance length, ANE)
5. HiFi-GAN vocoder (matcha_vocoder.mlmodelc): mel → 24kHz float32 PCM (ANE)

Command-Line Parameters:

Runtime Commands (Matcha-TTS engine cmd port 13176):

---

7. Moshi Service (bin/moshi-rag-service — Pure Rust Standalone Architecture)

WARNING: The old C++ moshi-service.cpp and two-component architecture (bin/moshi-service + WebSocket/Opus) are fully DEPRECATED, disabled, and removed from the active pipeline. All features are consolidated in the pure Rust standalone service (bin/moshi-rag-service).

Real-time speech-to-speech conversation using the MoshiRAG and Moshi models. Pure Rust implementation integrates directly with the G.711 / RTP pipeline via the custom Prodigy Bridge protocol without any WebSocket, Opus encoding/decoding, or double-buffering overhead.

Architecture:

• Single-Component: Run entirely within bin/moshi-rag-service (built from moshi-rag/rust/moshi-backend).
• Direct Pipeline Integration: Custom Prodigy Bridge listens for OAP/IAP and SIP commands, converting audio sample rates (24kHz model Mono <-> 8kHz G.711 PCMU) and routing caller logs directly.
• Asynchronous RAG Retrieval: Fully supports asynchronous patient/knowledge context injection from TOMEDO_CRAWL_SERVICE when a query requires factual grounding.
• Multi-language Support: Fully automated and configured via the database and --backend-config args routing to language-specific JSON configuration files.

Key fixes applied:

• matmul_dtype() returns BF16 on Metal (30× speedup for Q8 GGUF models on Apple Silicon).
• Highly-optimized streaming step intervals with Metal Warm-up caching.
• Custom ASR/STT diagnostic logging inside the Rust engine (asr.rs and stream_both.rs).
• Long-session conversation timeout handling (up to 60 minutes) to prevent premature disconnection on slow hardware.

Command-Line Parameters:

---

8. Outbound Audio Processor (bin/outbound-audio-processor)

Converts 24kHz float32 PCM from the TTS stage into 160-byte G.711 μ-law frames for the SIP client. Maintains constant 20ms output cadence.

Signal chain (per call):

1. DC blocking (first-order high-pass): α = 0.9947697 (~20Hz cutoff). Removes DC offset and LF rumble. Initialized with the first sample value to avoid onset click.
2. Presence boost (optional, default OFF): High-shelf biquad IIR filter, +3dB shelf at 2500Hz (Audio EQ Cookbook, S=1). Adds air/clarity to the telephone band.
3. Anti-aliasing FIR (63-tap, Hamming-windowed sinc): Cutoff 3400/12000 (normalized). ~43dB stopband attenuation. Coefficients computed once at startup, shared across all calls. Per-call fir_history[31] preserves filter state across chunks.
4. 3:1 Decimation: Keep every 3rd filtered sample (24kHz → 8kHz).
5. G.711 μ-law encode (ITU-T compliant): ULAW_CLIP=32635, ULAW_BIAS=132. Encodes int16 PCM to 8-bit μ-law byte.

Output scheduler: Dedicated sender thread fires every 20ms using steady_clock. Sends exactly 160 bytes per tick. If the TTS buffer is empty (silence, no engine docked, or a FLUSH_TTS just drained residual PCM during an engine swap), sends 0xFF (μ-law silence) to maintain RTP clock continuity. Scheduler resync guard: if OS sleep/load spike causes >100ms drift, snaps next_tick to now instead of firing a burst of catch-up frames.

SPEECH_ACTIVE handling: Clears all per-call buffers and resets FIR/DC/biquad state immediately when VAD signals caller speech. A configurable sidetone guard (default 1500ms) suppresses flushes arriving shortly after new TTS audio — prevents echo from triggering a spurious flush.

WAV recording (optional): When enabled, records the 8kHz int16 PCM output per call. Written to disk on CALL_END.

Command-Line Parameters:

Runtime Commands (cmd port 13152):

---

9. Frontend (bin/frontend)

Central control plane. Serves the web UI, manages service lifecycles, aggregates logs, and exposes all configuration via REST API.

Storage: SQLite database (whispertalk.db) — persists service configurations, log level settings, and test results.

Log aggregation: Each service sends structured log entries as UDP datagrams to port 22022. Frontend stores them in SQLite (ring-buffered in memory for fast /recent queries) and streams them live via SSE.

Full HTTP API (port 8080):

Web UI features:

• Service management: start/stop/restart each pipeline service independently
• Real-time log streaming with per-service and per-level filtering
• Log level control: checkboxes (ERROR/WARN/INFO/DEBUG/TRACE) applied immediately and persisted
• VAD configuration: threshold, silence duration, max chunk length — runtime update without restart
• Whisper configuration: model selection, hallucination filter toggle
• Kokoro configuration: synthesis speed slider, SYNTH_WAV test, neural G2P and language selection
• VITS2 / Matcha configuration: voice, G2P backend, language dropdowns with Save; engines are greyed out when no compatible model is installed for the selected language
• OAP configuration: WAV recording toggle + directory, presence boost toggle
• SIP management: add/remove SIP lines, view RTP statistics
• Beta testing page: audio injection into live calls via Test SIP Provider
• Test infrastructure: ASR accuracy tests, pipeline WER tests, LLaMA quality tests, codec quality tests
• tomedo-crawl configuration: Tomedo server IP/port, mTLS certificate upload, Ollama subservice management, crawl schedule, vector store status

---

10. tomedo-crawl (bin/tomedo-crawl)

RAG sidecar that crawls a Tomedo EMR server and provides per-patient context to the LLaMA service.

Components:

• Tomedo crawler: fetches patient list, diagnoses, medications, appointments, and phone numbers via mutual TLS HTTPS.
• Vector store: hnswlib HNSW in-memory ANN index + SQLite persistence (encrypted with SQLCipher).
• Phone index: local SQLite table mapping digit-normalised phone numbers to patient IDs; enables sub-100 ms caller identification from a SIP phone number.
• Ollama client: calls POST /api/embeddings to generate float32 embeddings for each text chunk.
• HTTP API (port 13181): serves /health, /query, /caller, /crawl/trigger, /ollama/*, /config.

Command-Line Parameters:

See docs/tomedo-crawl.md for the full API reference, database schema, Tomedo API details, and security model.

---

Interconnect Communication

All inter-service communication uses interconnect.h (a shared header, no external library):

• Management channel (base port +0): Typed control messages — CALL_START, CALL_END, SPEECH_ACTIVE, SPEECH_IDLE, PING/PONG
• Data channel (base port +1): Binary Packet frames — variable-length payloads tagged with call_id and PacketType (audio PCM, text, G.711)
• Command port (base port +2): TCP command interface, one connection per request, 10s recv timeout
• TCP_NODELAY: Enabled on all connections for minimum latency
• Auto-reconnect: Downstream connections retry every 200ms until reachable; upstream server accepts reconnections at any time
• LogForwarder: Sends structured log entries as UDP datagrams to FRONTEND_LOG_PORT (22022)

---

Log Level Control

Every service supports 5 levels: ERROR, WARN, INFO, DEBUG, TRACE.

Three ways to set log level:

1. Startup argument: --log-level DEBUG
2. Frontend UI: Log level checkboxes — applied immediately to the running service, persisted in SQLite for restarts
3. Direct command: Send SET_LOG_LEVEL:DEBUG to the service's cmd port via TCP

---

Testing

Pipeline WER Test

python3 tests/run_pipeline_test.py <MODEL_NAME> [TESTFILES_DIR]

Injects WAV samples through the full pipeline via Test SIP Provider, collects Whisper transcriptions from the frontend log API, and computes character-level similarity against ground truth.

• PASS: ≥ 99.5% similarity
• WARN: ≥ 90% similarity
• FAIL: < 90% similarity

Test samples: Testfiles/sample_NN.wav + sample_NN.txt pairs.

Test SIP Provider (bin/test_sip_provider)

B2BUA test tool that injects audio files into the pipeline as if they were real phone calls. Supports WAV recording of both legs of each conference call.

./test_sip_provider --port 5060 --http-port 22011 --testfiles-dir Testfiles

HTTP API (port 22011):

Audio Quality Collection (tests/run_stage7.py)

End-to-end diagnostic script:

python3 tests/run_stage7.py [--iterations N]

Starts all services, connects test calls, enables WAV recording, injects samples, collects logs, and saves WAV files from both OAP and Test SIP Provider. Produces stage7_output/run_N/ directories with pipeline.log, oap_call_*.wav, and tsp_call_*.wav.

Unit/Integration Tests

./runmetobuildeverything
cd build && ctest --output-on-failure

Tests are built by default (BUILD_TESTS=ON). To skip them: ./runmetobuildeverything --no-tests

Test binaries: test_sanity, test_interconnect, test_kokoro_cpp, test_integration.

---

Whisper Model Benchmarks

Hardware: Apple M4, macOS 25.2.0
whisper.cpp: v1.8.3 (CoreML + Metal)

whisper-cli Direct Test (5 samples, clean 16kHz PCM)

All model/backend combinations achieved 5/5 perfect transcription on clean input.

Full Pipeline Test (20 samples, G.711 μ-law via SIP/RTP)

Audio path: WAV → 8kHz μ-law G.711 → RTP → SIP Client → IAP (8→16kHz) → Whisper

* CoreML warmup caused first-inference timeout
** Sample_01 failed (41.8%) due to CoreML warmup causing VAD to miss first half

Scoring: PASS = ≥99.5% similarity, WARN = ≥90%, FAIL = <90%

Recommendations

• Accuracy priority: large-v3 + Metal — 1627ms avg, no warmup delay, best accuracy
• Speed priority: large-v3-turbo + CoreML — 688ms avg after initial warmup

Key Findings

1. Quantization (q5_0): negligible accuracy impact vs. full-precision models
2. CoreML warmup: 20–35s first-inference compilation cost, one-time per service lifetime
3. Turbo trade-off: ~2× faster, slightly more WARN results. 4-layer decoder occasionally misses nuances in G.711-degraded audio.
4. Consistent failures: some samples fail across all model configs due to G.711 codec artifacts, not model limitations