Medical STT Benchmark

Evaluation framework for speech-to-text models on medical conversation data.

Full write-up & leaderboard: omi.health/research/stt-benchmark Built by Omi Health · All research · Omi Scribe

Leaderboard

Dataset: PriMock57 (55 files, ~80,500 words) | Models: 42 comparable single-stream models | Updated: 2026-04-29

Ranked by Canonical Medical WER (M-WER)

Ranked by Canonical M-WER v2. Avg Speed = wall-clock seconds per ~7.5 min file (lower is better; not normalized for hardware tier — H100 ≫ A10 ≫ T4). Type: API (cloud), T4/A10/H100 (NVIDIA GPU tier via NeMo/vLLM/transformers), Apple Silicon (MLX/Native on M-series). Additional metrics in results/metrics/{model}_medical_wer.json.

*Google Gemini 3 Pro Preview completed 54/55 comparable files.

^Gemma 4 models use 30s chunking (model max audio = 30s).

Chunking Strategy

Most cloud APIs and native long-form models are evaluated on full audio. Chunking is only used when a model has an audio-length, token, memory, or decoder-behavior constraint.

The shared helpers live in transcribe/chunking_utils.py:

• Post-hoc overlap + LCS merge: Canary-Qwen, Canary Flash, Granite, and Azure Phi-4 use overlapping chunks and text-level LCS merging.
• CTC/HF chunking: MMS uses HF pipeline chunking with character timestamps. MedASR defaults to a stronger KenLM-backed short-chunk mode (8s chunks, 1s overlap) while preserving the old HF 20s/2s baseline via --decode_mode hf_pipeline.
• Simple concat: Gemma 4 keeps non-overlapping 30s chunks because overlap/context merging tested worse for that model.
• No chunking: Qwen3-ASR, cloud batch APIs, and native long-form paths keep full-audio decoding when the model supports it.

MedASR chunk/overlap experiments can be run with:

python scripts/run_medasr_chunk_ablation.py --audio_dir data/raw_audio --include_hf_baseline --evaluate

Multi-speaker models (separate — different metric)

These models output per-speaker transcripts with diarization, evaluated with cpWER (concatenated permutation WER) instead of standard WER. Not directly comparable to the single-stream leaderboard above.

The model outputs SegLST segments (speaker-tagged words with timestamps) via SpeakerTaggedASR. On well-diarized files (30/55), per-speaker WER is competitive with single-speaker Parakeet models. On 12/55 files diarization fails — the model assigns text to the wrong speaker or merges both into one stream. This is an early streaming model (v1) optimized for real-time display.

Per-speaker references: PriMock57 TextGrid files (doctor + patient separately with timestamps).

Metrics explained

Additional metrics available per model in results/metrics/{model}_medical_wer.json: M-CER (character-level error on medical substitutions), Token Recall (occurrence-weighted), Entity Recall (binary), per-category M-WER breakdown (drugs, conditions, symptoms, anatomy, clinical).

Canonical M-WER v2

Medical scoring uses the same base text normalization as standard WER, then applies a narrow clinical canonicalization pass before medical-term alignment:

• Corrects known reference-side PriMock57 spelling issues: paracetemol → paracetamol, thyrocsin → thyroxine, flem → phlegm.
• Collapses accepted word-boundary variants such as water works/waterworks, hay fever/hayfever, and straight away/straightaway.
• Normalizes low-risk morphology and symptom wording variants such as tests/test, headaches/headache, itchy/itching, and tummy/stomach.
• Keeps clinically meaningful or unsafe substitutions penalized, including hyperthyroidism/hypothyroidism, modulite/modulate, clenil misspellings, ventolin/ventilin, and implanon/implant.

This canonicalization is used only for medical metrics. Overall WER remains the standard normalized WER.

Quick Start

# Install Git LFS (required for audio files)
git lfs install

# Clone and install
git clone https://github.com/Omi-Health/medical-STT-eval.git
cd medical-STT-eval
pip install -r requirements.txt

# Add API keys
cp .env.example .env

# Run transcription (outputs to results/transcripts/)
python transcribe/groq_whisper_transcribe.py --audio_dir data/raw_audio

# Generate metrics
python evaluate/metrics_generator.py --model_name groq-whisper-large-v3
python evaluate/medical_wer.py --model groq-whisper-large-v3 --output results/metrics/groq-whisper-large-v3_medical_wer.json

# Update leaderboard
python evaluate/comparison_generator.py

Note: Existing per-model transcripts are tracked in results/transcripts/ for reproducibility. To re-evaluate or add a new model, run the relevant transcription script then metrics_generator.py and medical_wer.py.

Project Structure

medical-stt-benchmark/
├── data/
│   ├── raw_audio/              # 57 WAV files (Git LFS)
│   └── cleaned_transcripts/    # 57 reference transcripts
├── transcribe/                 # Model-specific transcription scripts + base class
│   └── base_transcriber.py     # Shared base (loads .env)
├── evaluate/                   # Evaluation scripts
│   ├── text_normalizer.py     # Custom WER normalizer (see Metrics below)
│   ├── clinical_canonicalizer.py # Canonical M-WER v2 mappings
│   ├── wer_calculator.py
│   ├── metrics_generator.py
│   ├── medical_wer.py
│   └── comparison_generator.py
└── results/
    ├── metrics/                # WER, medical WER, and speed JSON
    ├── comparisons/            # leaderboard.json, per_file_results.json
    └── transcripts/            # tracked model transcripts for reproducibility

Public vs Private Contents

The public repository contains reproducible benchmark code, reference transcripts, tracked model transcripts, metrics, and leaderboard outputs.

A local private_analysis/ directory may exist in development checkouts. It is intentionally git-ignored and is not part of the public benchmark. Public documentation and reproducible workflows should not depend on it.

Supported Platforms

Adding a New Model

1. Create transcribe/your_model_transcribe.py inheriting from BaseTranscriber
2. Implement transcribe_file() returning transcript text
3. Run on dataset: python transcribe/your_model_transcribe.py --audio_dir data/raw_audio
4. Generate metrics: python evaluate/metrics_generator.py --model_name your-model
5. Generate medical metrics: python evaluate/medical_wer.py --model your-model --output results/metrics/your-model_medical_wer.json
6. Update comparisons: python evaluate/comparison_generator.py

Dataset

PriMock57: 57 doctor-patient consultations, 81,236 words of British English medical dialogue. 55 files used for evaluation (2 excluded due to processing issues).

Audio files are tracked with Git LFS. Reference transcripts derived from PriMock57 under CC BY 4.0.

Citation:

@inproceedings{korfiatis2022primock57,
  title={PriMock57: A Dataset Of Primary Care Mock Consultations},
  author={Papadopoulos Korfiatis, Alex and Moramarco, Francesco and Sarac, Radmila and Savkov, Aleksandar},
  booktitle={Proceedings of the 60th Annual Meeting of the ACL},
  year={2022}
}

Metrics

• WER: Word Error Rate (lower is better)
• M-WER: Canonical Medical WER v2 on medical terms only (lower is better)
• Drug M-WER: Canonical M-WER v2 on drug terms only (lower is better)
• Accuracy: 1 - WER
• Speed: Average seconds per ~7.5 min file

Text Normalization

WER is calculated using a custom normalizer (evaluate/text_normalizer.py) based on Whisper's EnglishTextNormalizer with two fixes:

1. "oh" bug fix: Whisper treats "oh" as the digit zero (self.zeros = {"o", "oh", "zero"}). In medical conversations "oh" is always an interjection. Our normalizer removes "oh" from this set.
2. Word equivalence mappings: ok/okay/k, yeah/yep/yes, mum/mom, alright/all right, kinda/kind of — variant spellings that Whisper does not normalize to the same form.

This reduced WER by ~2-3% across all models compared to stock Whisper normalization. No runtime dependency on the openai-whisper package.

Citation

If you use this benchmark, please cite:

APA — Omi Health. (2025). Benchmarking Speech-to-Text Models for Long-Form Medical Dialogue. https://omi.health/research/stt-benchmark

BibTeX

@misc{omi_stt_benchmark_2025,
  title   = {Benchmarking Speech-to-Text Models for Long-Form Medical Dialogue},
  author  = {{Omi Health}},
  year    = {2025},
  url     = {https://omi.health/research/stt-benchmark},
  note    = {Medical Word Error Rate (M-WER) leaderboard, 42 models}
}

License

MIT License. Dataset under CC BY 4.0.

---

Built by Omi Health — the private AI stack for healthcare.