voice-evals

Offline, reference-free evaluation of the audio experience of AI voice calls — how the agent sounds and behaves on the line, not what it said. It scores the audio bytes themselves, so it catches what transcript/tool-call evals can't: robotic synthesis, dropouts, sluggish replies, awkward silences, failed barge-in, and the #1 production failure — the agent going dead when background noise jams its turn detection.

Two ways to use it:

1. Score a folder of production recordings — drop .mp3/.wav in, get per-clip scores, an HTML report, Parquet/JSON, and pytest regression gates.
2. Run a live eval set against a phone number — a persona caller-bot places real Twilio calls to your agent, drives hard-caller scenarios (interruptions, café noise, angry callers…), and aggregates the scores into one report — pre-merge or cross-vendor A/B signal on the experience, not correctness.

Design principle: everything is reference-free / unsupervised. No per-clip ground-truth labeling anywhere in the core pipeline. The only human labeling is an optional ~30–50 clip calibration set, used once to validate the judge.

Scope: this repo scores the audio experience only. Information correctness and tool-call success are owned by your text/OTel evals. A scenario's goal exists to make the simulated caller behave realistically (and get frustrated when mishandled), not to score task completion here.

---

The three layers

Layers A and B run with zero API keys and zero network. Layer C is additive and off by default (scorers.judge: []); enable it with a config profile + a key.

What it measures, in plain language

The report speaks in concepts a non-technical reader understands; the raw metric names below are what technical users debug with (surfaced only in expanders).

---

Install

Python 3.11+ and ffmpeg on PATH. Base + [acoustic] + [dynamics] run with no GPU and no API keys.

# with uv (recommended)
uv sync --extra acoustic --extra dynamics                 # local-only core
uv sync --extra telephony --extra judge-openai            # + live calls + OpenAI judge
uv sync --all-extras                                      # everything + dev/test
uv run voice-evals --help

# or with pip
pip install -e ".[acoustic,dynamics]"
pip install -e ".[all]"

Most model weights self-fetch on first use (UTMOS via torch.hub, SQUIM via torchaudio, silero on first run, DNSMOS bundled). NISQA has no clean PyPI package and is vendored once:

python scripts/fetch_models.py nisqa      # clones NISQA + weights into .cache/models/

Secrets go in a gitignored .env (see .env.example) — never in YAML:

OPENAI_API_KEY=sk-...              # OpenAI judge + the simulated caller (STT/LLM/TTS)
GEMINI_API_KEY=...                 # (alt judge)
TWILIO_ACCOUNT_SID=AC...           # live telephony
TWILIO_AUTH_TOKEN=...
TWILIO_FROM_NUMBER=+1...            # your Twilio caller-ID number
TWILIO_AGENT_NUMBER=+1...           # the agent's number to dial (overridable with --to)
PUBLIC_STREAM_URL=wss://<tunnel>/caller   # your ngrok wss → localhost:8080

---

Quickstart A — score existing recordings

cp /path/to/*.mp3 corpus/                              # + optional <stem>.json sidecars
uv run voice-evals run --config config/example.local.yaml    # Layers A+B, no network
open outputs/runs/<run_id>/report.html

…with the audio judge on those recordings

example.local.yaml keeps the judge off. To score the same folder with the native-audio judge — the same judge as the live eval set, just pointed at your production recordings — enable it with a judge profile and a key (audio is sent to the API, so this is explicit opt-in):

export OPENAI_API_KEY=sk-...          # (or GEMINI_API_KEY) — or keep it in .env
uv run voice-evals run --corpus corpus/ --config config/judge.openai.yaml
open outputs/runs/<run_id>/report.html

Two judge knobs matter for production recordings:

• judge.judged_channel — agent_only grades just the agent's voice (needs a channel_map in the sidecar to isolate it); mono grades the whole call (caller + agent — flow, frustration handling). judge.openai.yaml uses mono.
• judge.exclude_if_model_family_matches — if a recording's tts_provider is the judge's own model family (e.g. an OpenAI-voiced agent graded by the OpenAI judge), the judge is skipped to avoid self-enhancement bias; set it false to force it. judge.openai.yaml sets it false.

For Gemini instead of OpenAI, use a config with judge.backend: gemini, judge.model: gemini-2.5-pro, and GEMINI_API_KEY set. --no-judge turns the judge off on any run. (Dynamics like barge-in/stall need an event log or split channels, so on bare recordings they stay sparse — see the channel question; the judge and acoustic layers work on any audio.)

Quickstart B — live eval set against your agent's phone number

# 1. Your agent is reachable at a phone number (deployed, or local dev behind a Twilio number).
# 2. Run ngrok → localhost:8080 and set PUBLIC_STREAM_URL to its wss URL in .env.
# 3. Smoke-test one call per scenario, then the full set:
uv run voice-evals call --scenarios scenarios/eval_set.yaml --config config/judge.openai.yaml \
  -k 1 --agent-version openai-rt --preamble 35        # 7 calls
uv run voice-evals call --scenarios scenarios/eval_set.yaml --config config/judge.openai.yaml \
  -k 3 --agent-version openai-rt --preamble 35        # 21 calls → one aggregated report

You set up no Twilio infrastructure — voice-evals call places the outbound call via Twilio's REST API and embeds the media-stream websocket server (binds 127.0.0.1:8080 for the duration of the call). You just run ngrok pointed at 8080.

Other commands:

uv run voice-evals ingest --corpus corpus/            # decode/normalize only; warm the cache
uv run voice-evals simulate --scenarios scenarios/example.yaml --mock --baseline-version main   # synthetic A/B, no telephony
uv run voice-evals report --run recordings/<ts>/report        # regenerate report.html from run.json (no re-calling)
uv run voice-evals run --baseline previous --strict           # exit 1 on any gate violation
uv run voice-evals calibrate --golden calibration/golden_set.csv
uv run voice-evals cache clear [--scorer dnsmos]
uv run voice-evals list-scorers

---

The eval set

scenarios/eval_set.yaml is a curated set of hard callers, each a distinct failure mode mined from real bad calls:

call runs -k calls per scenario (calls are stochastic — the caller LLM and the real agent both vary — so gate on the aggregate, not one flap). Each call is a distinct clip; scores roll up per scenario and overall into one report. Tag each run with --agent-version so the baseline: previous regression gate diffs one build (or vendor) against the last.

A scenario is a goal + persona + optional barge_in / background policy:

- id: noisy_cafe_barging_impatient_combo
  goal: "get the front gate access code reset because the caller is locked out"
  persona: { name: locked_out, style: "loud, clipped, impatient", patience: low,
             emotional_arc: "flares every time the agent goes quiet", adversarial: true, voice: verse }
  max_turns: 5
  barge_in:   { enabled: true, after_agent_s: 1.5, max_barge_ins: 3 }   # caller talks OVER the agent
  background: { enabled: true, gain_db: -16, voices: [onyx, fable] }     # intelligible café chatter on the open mic

Background noise (the #1 issue)

background: mixes synthesized intelligible voices continuously onto the caller's (open) mic. This reproduces the top production failure: babble jams the agent's endpoint detection so it can't tell the caller stopped, and it freezes — captured by stall_rate / latency while the agent's own voice quality stays clean (the babble is on the caller channel). gain_db: -20 ≈ realistic café; -14 is loud.

Barge-in

barge_in: makes the caller talk over the agent once it's spoken for after_agent_s. The barge_in scorer measures whether the agent yields, how fast (stop-latency p50/p95), the success rate, and false alarms — real barge-in over real telephony, which half-duplex simulation can't see.

---

Real telephony: setup & operating notes

voice-evals call dials the agent (--to / TWILIO_AGENT_NUMBER) with TwiML <Connect><Stream url=PUBLIC_STREAM_URL>, runs the persona caller full-duplex over the μ-law/8kHz audio, records both legs + an event log, and scores it. What you run:

• Your agent at a phone number (deployed, or local behind a Twilio number + its own webhook).
• ngrok → localhost:8080, with PUBLIC_STREAM_URL = its current wss://…/caller.

The reactive caller

The simulated caller is a cascade: STT of the agent's turns (gpt-4o-transcribe) → LLM reply (gpt-4o) → TTS (gpt-4o-mini-tts), streamed as μ-law. It hears and responds to the agent — answers questions, escalates when stalled. --no-stt makes it deaf (cheaper, pure barge-in timing). We deliberately kept the caller a controlled cascade rather than an audio-to-audio model so barge-in timing stays deterministic and runs stay reproducible; the caller is pluggable (CallerBot) if you want to swap it.

Deployed-agent preamble

Many deployed agents play an IVR / greeting / recording caveat before the real agent speaks. Pass --preamble N (or telephony.preamble_s) so the caller ignores the agent's first N seconds for turn-taking (it's still recorded) and engages only with the real agent. A deadlock guard opens the call if the greeting is missed.

Troubleshooting

voice-evals call writes each run to its own dated dir under recordings/<timestamp>/ (a durable library — never auto-wiped; delete by hand). One failed call is logged and skipped, not fatal; if every call fails it exits cleanly instead of scoring nothing.

---

The report

Written for a non-technical reader, debuggable by a technical one. Regenerate any run's report with voice-evals report --run <dir>/report (no re-calling).

1. Verdict + plain summary — "N of M scenarios passed across K calls."
2. What went wrong — plain-language findings, [Scenario] — [Concept]: scored X (aim Y) with a "Why it matters" line, worst first. No raw-metric dumps.
3. Scenarios (the hero) — a card per scenario: readable title, what it tests, PASS/WARN/FAIL, and concept tiles showing the value, the target, and pass/fail color.
4. What we measure — concept cards defining each measure + why it matters, with the raw metrics tucked in an "underlying measures" expander.
5. Scenario detail — each titled, with the judge's per-call reasoning + audio.
6. Technical details (collapsed) — threshold violations, regressions, run info.

---

Conversational dynamics: the channel question

Layer B accuracy depends on channel separation. The dynamics scorers prefer, in order:

1. Gateway/sim event log in the sidecar → exact (source="events").
2. Isolated agent/caller channels → accurate VAD (source="channels").
3. Mono mix → VAD + diarization estimate (estimated=True); barge-in isn't measured on mono.

The call/simulate recordings carry both an event log and dual channels, so timing is exact. latency and stall are event-based, so background noise can't fool them — unlike audio-derived gap metrics, which noise masks.

Sidecar JSON (<stem>.json next to the recording)

{
  "call_id": "abc123", "agent_version": "leasing-ai-2.14.0", "prompt_version": "p-...",
  "channel_map": {"agent_channel": 0, "caller_channel": 1},
  "events": [
    {"kind": "user_speech_end", "t_s": 3.2}, {"kind": "stt_final", "t_s": 3.2},
    {"kind": "tts_first_audio",  "t_s": 4.1}, {"kind": "agent_tts_start", "t_s": 4.1},
    {"kind": "barge_in_detected","t_s": 6.05},{"kind": "agent_interrupted","t_s": 6.25}
  ]
}

Latency reads tts_first_audio − stt_final; barge-in reads agent_interrupted − barge_in_detected — both exact. All fields optional; with none, Layer A still works and Layer B falls back to VAD.

---

The audio judge (Layer C)

Sends the audio (not a transcript) to a native-audio model with a versioned rubric and scores nine dimensions 1–5 (5 = best), reasoning before each score:

naturalness · prosody · fluency · pace · responsiveness · emotional_alignment · intelligibility · conversational_flow · frustration_handling

plus a holistic overall, notable_timestamps, and a summary → flattened as judge_naturalness … judge_overall. Set judged_channel: mono to judge the whole conversation (caller + agent — flow and frustration handling), or agent_only to judge the agent's audio in isolation. The rubric/system prompt are plain text in config/; bump RUBRIC_VERSION on any change (it keys the cache + invalidates calibration).

⚠ The judge is latency-blind. An audio model doesn't experience real-time duration — it can rate responsiveness 5/5 on a call with 20s of dead air. Timing is owned by the deterministic layer (latency, stall); don't gate responsiveness on the judge. This is exactly why both layers exist.

Guards. exclude_if_model_family_matches stops the judge grading audio from its own model family (self-enhancement bias — critical for a cross-vendor A/B: an OpenAI judge inflates an OpenAI agent). Degenerate inputs (silence, no agent speech) surface as an error, never hallucinated numbers.

Cost/PII. ~32 tokens/s of audio ≈ $0.003–0.006 for a 60s clip; caching makes reruns free. Recordings contain resident data — hosted-judge use is explicit opt-in; use the local-only profile for CI and backend: local_omni (self-hosted Qwen-Omni) or Vertex-with-DPA for compliant paths.

---

Calibration (validate the judge before trusting it)

The only place humans label — optional, one-time. Fill calibration/golden_set.csv (clip_id, dimension, human_score) for ~30–50 clips using the rubric's anchors, then:

uv run voice-evals calibrate --golden calibration/golden_set.csv

Reports per-dimension quadratic-weighted Cohen's κ + correlation and recommends which dimensions to gate in CI (κ ≥ calibration.kappa_bar, default 0.6). Expect intelligibility/naturalness/fluency to agree well; treat low-κ dims as advisory.

---

pytest regression gates

uv run pytest -m "not slow and not judge"    # default CI: Layers A + B, no network/keys
uv run pytest -m judge                        # judge gates (needs a key or local model)

tests/eval_gates/ parametrizes one assertion per (clip, gated metric) plus a regression gate that fails when a metric degrades beyond max_delta vs the baseline.

---

Configuration

Profiles in config/:

• example.local.yaml — Layers A+B, judge off (CI-safe).
• default.yaml — production defaults, judge opt-in.
• judge.openai.yaml — the full live-eval profile: acoustic + dynamics (incl. stall)
  • OpenAI gpt-audio judge on the whole conversation, all metrics gated pass/fail.

Gates support absolute floors/ceilings (with soft warn) and regression deltas grouped by agent_version. Telephony knobs live under telephony: (preamble_s, connect_timeout_s, max_call_s, vad_rms, end_silence_ms, port). Thresholds are heuristic starting points — calibrate to your corpus. Full schema in src/voice_evals/config.py.

---

Repository layout

voice-evals/
├── config/            local/default/judge.openai YAML, rubric.default.txt, judge_system.txt
├── scenarios/         eval_set.yaml (the hard-caller set) + example/live_barge_in/noisy_cafe
├── corpus/            (gitignored) drop audio + optional .json sidecars here
├── recordings/        (gitignored) live-call library — one dated dir per `call` invocation
├── outputs/  .cache/  (gitignored) run artifacts + decoded-audio/result cache
├── calibration/       golden_set.csv (human labels for judge validation)
├── scripts/           fetch_models.py (vendor NISQA)
├── src/voice_evals/   models, config, ingest, cache, runner, aggregate, report, cli,
│                      scorers/{acoustic,dynamics(latency,turn_taking,barge_in,stall),judge},
│                      simulate/ (scenario, orchestrator, openai_backends, gating,
│                      telephony + telephony_live — Twilio Media Streams caller)
└── tests/             unit tests + eval_gates/ regression suite

Non-goals (v1)

Not a live/streaming monitor, not an STT/correctness system, not a labeling UI. Offline batch over recorded files, plus on-demand live calls to build those recordings.