DeafBench synthetic-v2

DeafBench synthetic-v2 is a frozen, 25-sample benchmark specification for testing whether automatic speech recognition systems preserve information that matters in accessible captions. It keeps conventional word error rate separate from typed critical-information recall so a plausible transcript cannot hide a lost time, digit sequence, username, code, Wi-Fi name, or proper name.

This repository publishes the reference text and reproducibility metadata. It does not publish audio, validation speech, model predictions, benchmark run directories, or model weights.

Intended use

Use this repository to inspect the benchmark schema, build compatible tooling, or verify the hashes and evaluation contract for a local DeafBench run. The audio named by the frozen manifest is intentionally absent, so this repository alone cannot reproduce acoustic scores.

The benchmark is an accessibility-focused synthetic lane. It is not an Open ASR Leaderboard dataset, a Hugging Face leaderboard submission, a clinical evaluation, or proof that one model is generally better than another.

Benchmark scope

• 25 English synthetic reference samples
• typed accessibility-critical values
• controlled background scenes and generation records
• model-independent quality gates for regenerated samples
• a frozen faster-whisper baseline retained as historical corpus evidence

Only core-001, core-009, core-011, and core-016 were regenerated for synthetic-v2. The remaining 21 samples retain their core-v1 parent audio hashes. References and critical-entity labels remain unchanged across all 25 samples.

Data fields

Each row in references.jsonl contains:

The typed evaluator uses exact or narrowly canonical comparison according to entity type. It does not fuzzy-match usernames, codes, passwords, Wi-Fi names, or different proper names.

Included files

• references.jsonl: the 25 reference rows exposed by the dataset viewer
• generation-manifest.jsonl: parentage, synthesis, scene, and hash metadata
• quality-policy.json: predeclared, model-independent acceptance gates
• quality-report.json: gate results and supporting validation evidence
• freeze-manifest.json: the frozen corpus, model, evaluator, and result record
• DATA_MANIFEST.json: hashes tying these Hub files to their GitHub source

Audio paths and audio hashes appear in the manifests as reproducibility evidence, but the media itself is not redistributed here.

Metrics

DeafBench reports conventional ASR metrics and its accessibility layer as separate metric families:

• orthographic WER and normalized WER
• character error rate (CER)
• substitutions, insertions, and deletions
• strict lexical critical-information recall
• canonical semantic critical-information recall
• critical failures by entity type
• real-time factor (RTFx), latency, and peak VRAM where measured

The frozen faster-whisper synthetic-v2 evidence records 25.2% WER, 69.4% strict lexical recall, 96.8% canonical semantic recall, and 35 substitutions, 6 insertions, and 31 deletions. This is historical evidence for this corpus and configuration, not a current rerun or a leaderboard claim.

Reproducibility

The source artifacts are pinned to DeafBench Git commit aa1bb5fbde579b303022a315062567bfabb95aa2. Verify each exported file against DATA_MANIFEST.json, then use freeze-manifest.json to verify the evaluator, model revision, decoding configuration, corpus hashes, and dependency versions.

The complete evaluator and benchmark documentation live in the DeafBench repository. The public project page explains the evidence lanes at 488315.github.io/products/deafbench.

Limitations

• The corpus is small, synthetic, and English-only.
• It does not represent Deaf and hard-of-hearing speech diversity.
• It does not establish demographic fairness or clinical suitability.
• The Hub repository omits audio, so acoustic results require the separately authorized local corpus.
• Local latency, RTFx, and VRAM depend on hardware and runtime configuration.
• Synthetic-v2 results cannot be compared directly with Open ASR Leaderboard macro WER or with results from different datasets and normalization rules.

Licensing and attribution

The material published in this repository is licensed under Apache-2.0. Model names and revisions are factual reproducibility metadata; no third-party model weights or restricted benchmark media are included. See LICENSE for the full license text.

Citation

Please cite the DeafBench GitHub repository and include the benchmark version, source commit, corpus manifest hash, evaluator revision, model revision, and decoding configuration with any reported result.