Pause-Tolerant Endpointing Benchmark

An open benchmark for a specific way voice agents fail: deciding the caller has finished talking when they were only pausing between chunks of a number.

Plus four baseline endpointing strategies, a trained semantic classifier, and a fused endpointer that beats pure silence detection on the failure that matters.

Everything runs offline on CPU. No API keys, no accounts, no GPU.

---

The problem

Voice agents decide when to reply using endpointing: notice the user has gone quiet for N milliseconds, assume the turn is over, respond. Tune N to ordinary conversation and you land somewhere around 500–700 ms.

Then someone reads out a member ID:

"My member ID is four seven two ... [700ms] ... nine one ... [600ms] ... six three"
                                    ^^^^^^^^
                                    the agent interrupts here

People chunk numbers when they read them. A 700 ms threshold fires inside the first pause and the agent talks over the caller — who now has to start the whole number again. Raise the threshold to survive that, and every ordinary "yes, that's right" gains a second of dead air.

That is the tradeoff this repo measures.

The headline result

Read the chart as: down is fewer interruptions on chunked numbers, left is less dead air on ordinary turns, and the green box is where a shippable system would have to live. The red, blue and green curves are the same three silence-based strategies swept across their thresholds. None of them enters the box. They cannot: moving down the curve is moving right.

At the same latency as the fastest VAD, adding a "is this thought finished?" classifier drops mid-number interruptions from 100% to 18% and backchannel false-ends from 100% to zero.

It is not free, and the last column is why: the gate converts a loud failure (talking over the caller) into a quiet one (never answering). That is probably the right trade for a voice agent, but it is a trade, not a fix. A variant with a safety valve removes the no-response failure entirely and instead pays a latency tail — 6.7% cut-off, 350 ms median, 1252 ms p90. Neither configuration reaches the green box in the chart.

Full numbers, caveats, and the things that went wrong: FINDINGS.md.

Method

Phase 1 — a synthetic corpus with exact pause structure. Real call recordings are not available, so the dataset is generated: 600 clips, 16 kHz, ~71 minutes, three Piper TTS voices. Each clip is built by synthesising speech chunks separately, trimming them, and splicing silences of exact duration between them — so the ground-truth endpoint and all 660 internal pause boundaries are known to the sample rather than annotated. validate_dataset.py re-measures every clip off the waveform and confirms the labels agree to within 20 ms.

Five categories, 120 clips each:

Phase 2 — four strategies behind one streaming interface. Every endpointer consumes 10 ms frames in order and returns True on the frame where it declares the turn over. Nothing looks ahead.

• A WebRTC VAD, silence threshold swept 300–1200 ms
• B Silero VAD, same sweep
• C Whisper (faster-whisper base) transcript stability + a short silence threshold — what most production stacks ship
• D Oracle, reads the label. The ceiling.

Phase 3 — metrics that admit two failure modes. Cut-off rate and latency move in opposite directions, so neither is reported alone, and latency is p50/p90/p95 rather than a mean — a mean launders exactly the long waits that make an agent feel broken.

Phase 4 — the semantic gate. A DistilBERT binary classifier trained on Phase 1 transcripts truncated at word boundaries: full transcript → complete, truncation → incomplete, backchannel token → not a turn end.

Two deliberate choices there. Negatives are all chunk-boundary prefixes plus three sampled word-boundary truncations per clip, rather than every possible truncation — taking all of them pushes the class ratio past 13:1 on numeric clips and fills the training set with prefixes no endpointer is ever queried on, while the chunk boundaries are precisely the states a real pause lands in. And backchannel tokens are labelled not a turn end even though "okay" is a grammatically complete utterance; the question being asked is "should the agent reply", not "is this a sentence".

Strategy E fires only when Silero reports silence and the classifier says the thought is finished:

acoustic (Silero)     -> "the user has gone quiet"
semantic (DistilBERT) -> "what they said is a finished turn"
                         both must agree, or keep waiting

Splits are by clip, so a clip's own truncations cannot leak across the boundary, and all five strategies are scored on the same 300 held-out clips.

Reproducing

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
./run_all.sh

Everything is seeded (--seed 1234) and regenerates identically. The full run takes roughly an hour on an 8-core CPU with no GPU, most of it precomputing Whisper prefix transcripts. First run also downloads Piper voices (~190 MB), silero-vad, faster-whisper base, and distilbert-base-uncased; after that nothing touches the network.

For a smoke test, --quick uses 30 clips per category and one training epoch, and finishes in about 15 minutes:

./run_all.sh --quick

--quick checks that the pipeline runs, not that the findings hold. One epoch on a fifth of the data leaves the classifier undertrained — strategy E scores far worse there than in the full run, which is a property of the smoke test and not a result. Only the full run reproduces the numbers in FINDINGS.md, and step 6 checks that it did.

DATA_DIR, RESULTS_DIR and MODEL_DIR can be pointed at a scratch tree so a smoke run does not overwrite a completed one:

DATA_DIR=data_quick RESULTS_DIR=results_quick MODEL_DIR=models/quick ./run_all.sh --quick

Note that run_all.sh regenerates the dataset from scratch, which clears the transcript cache with it. If a run is interrupted during the Whisper step, re-running precompute_transcripts.py on its own resumes from what is already cached; re-running run_all.sh starts that step over.

Layout

generate_dataset.py        Phase 1  synthesise + splice the corpus
validate_dataset.py        Phase 1  re-measure every clip against its label
precompute_transcripts.py  Phase 2  cache Whisper prefix transcripts for C and E
run_baselines.py           Phase 2  score every strategy -> results/raw_detections.csv
metrics.py                 Phase 3  rates, latencies, tradeoff chart
train_classifier.py        Phase 4  DistilBERT (+ TF-IDF control) completion classifier
check_findings.py          re-checks every number quoted in the docs against the CSVs
run_all.sh                 end to end

endpointing/
  audio.py         resample, trim, splice, noise floor
  corpus.py        the text material for all five categories
  tts.py           Piper wrapper with an on-disk chunk cache
  endpointers.py   Endpointer interface + strategies A, B, C, D
  fused.py         the classifier wrapper and strategy E
  textprep.py      normalisation and training-example construction

results/
  tradeoff.png              the headline chart
  curves.png                per-category cut-off and latency vs threshold
  confusion_*.png           classifier confusion matrices
  summary_table.md          full results table, markdown
  results_by_category.csv   every strategy x category
  results_overall.csv       every strategy, pooled
  classifier_metrics.csv    classifier metrics incl. numeric/non-numeric split

Reading order

If you want the ideas in the order that makes them land:

1. FINDINGS.md — the conclusions, including what did not work.
2. results/tradeoff.png — the whole problem in one image. Understand why the curve has that shape and you understand endpointing.
3. generate_dataset.py — the pause structures encode what actually goes wrong on real calls.
4. metrics.py — why cut-off rate and latency have to be reported together, and why p90 matters more than the mean.
5. endpointing/fused.py — the architectural idea worth carrying: acoustic signal answers is there sound, semantic signal answers is the thought finished. Two questions, two models, one fused decision.

The one-paragraph version

A silence threshold is a single number being asked to encode a judgement about language. It works when the caller says "yes" and fails when the caller says a member ID, because the acoustic evidence in both cases is identical — a person who stopped making noise — and the difference between them is not acoustic at all. You cannot fix that by tuning the number, only by moving along a tradeoff curve, which is why the chart above has the shape it does. What you do about it is ask a second question the audio cannot answer: given the words so far, is this a finished thought? Then require both answers before you speak.

Notes on the environment

• setuptools is pinned <81 because webrtcvad imports pkg_resources, which setuptools 84 removed.
• Piper was chosen over Coqui/pyttsx3: fully offline, deterministic for a fixed voice and config, and small enough to run fast on CPU.
• No install failures otherwise; the versions verified are in requirements.txt.