STT Bench — Greek Speech-to-Text Benchmark for Agricultural Voice Entry

A small evaluation harness that compares speech-to-text (STT) models on Greek agricultural speech, to decide which model to deploy for a voice-entry feature in AgroBox (a farm-management platform). Farmers speak a sentence like "έριξα 25 κιλά ουρία στο κάτω χωράφι" and the system should transcribe it correctly — domain terms, product names, quantities and all.

This repo is Part 1: picking the STT model with data, not gut feeling. Part 2 (extracting the transcript into structured form fields) builds on the winner.

One recording, three models side by side. Here Gemini gets the crop and the toponym right («τις ελιές στη Χαλκιδική»); whisper-small garbles them; qwen-0.6b collapses — and the timings (2.9 s / 5.4 s / 43.7 s) show the speed trade-off.

Results

17-phrase Greek agricultural test set, three models, measured end-to-end:

_{WER = Word Error Rate after normalization (lower is better). RTF = Real-Time Factor = processing time ÷ audio duration (below 1.0 = faster than real-time). * Whisper's mean latency is 4.6 s, skewed by a single outlier — it spent ~30 s on a noisy/laughing clip vs. ~3 s normally.}

Decision: Gemini. For a care-logging feature, accuracy on domain vocabulary (product names like Karate Zeon/Neogen, terms like «ουρία», quantities) is what matters — and that is exactly where Gemini pulls ahead (8.5% vs. 34%/62%). Its ~6 s latency is acceptable, with the trade-offs being network dependency and per-call cost. whisper-small is kept in mind as an offline fallback (fast, runs locally, but 4× less accurate and unstable on noise).

Key findings

• Domain vocabulary is the differentiator. All models handle plain words well; the gap opens on brand names and specialized terms. Gemini preserves Karate Zeon, Ridomil, Neogen; Whisper transliterates them («καρατε ζεον»); Qwen collapses — on one clip it even output Portuguese.
• Whisper (small) is fast and offline but fragile. Great real-time factor, but 34% WER on this domain and a ~10× slowdown on a non-speech (laughing) clip.
• Qwen3-ASR 0.6B on CPU is dominated — worst accuracy and slowest (RTF 4.0, ~45 s/utterance). A GPU and/or the 1.7B variant would change the latency story, not tested here.
• Normalization matters for a fair comparison. Before scoring, both reference and hypothesis are lowercased, stripped of accents and punctuation, and have spelled-out numbers mapped to digits — otherwise Gemini was unfairly penalized for writing «πενήντα» instead of «50».

How it works

browser mic ──▶ FastAPI /transcribe ──▶ ffmpeg (16 kHz mono WAV) ──▶ [ adapters ] ──▶ JSON
                                                                        │
                        WhisperAdapter · GeminiAdapter · QwenAdapter ───┘

Every model is wrapped in a common adapter interface — a class with a name and a transcribe(wav_path) -> str method. Adding a model is a one-line change to the model list; the endpoint and the batch runner don't change.

• app.py — FastAPI service: a browser UI (static/index.html) with a mic button, and a /transcribe endpoint that runs all adapters and returns their transcripts + timings.
• preprocessing.py — convert_to_wav() (ffmpeg): normalizes any input to 16 kHz mono WAV so every model gets identical input.
• adapters/ — one file per model, all sharing the same interface.
• evaluation/run_batch.py — runs every (clip × model) over the test set, with on-disk caching (md5 of clip+model) so re-runs are instant and don't burn API quota. Writes results/raw_results.json.
• evaluation/score.py — normalization + WER (via jiwer) and RTF per model.

Models compared

Test set

17 spoken Greek phrases (dataset/audio/), organized around the 8 care types of the target feature (irrigation, tillage, sowing, fertilization, organic fertilization, harvest, plant protection, general), deliberately covering: numbers + units, real commercial product names, toponyms, relative dates/times, disfluent "natural" speech (fillers, false starts), and a few noisy recordings. Ground truth lives in dataset/manifest.csv.

Running it

Requirements: Python 3.11, ffmpeg on PATH, and a Gemini API key.

python -m venv .venv
.venv\Scripts\Activate.ps1          # Windows PowerShell  (source .venv/bin/activate on *nix)
pip install -r requirements.txt

# secrets — never commit this file (it is git-ignored)
echo GEMINI_API_KEY=your_key_here > .env

The first run downloads model weights (~0.5 GB Whisper, ~1.5 GB Qwen) into the Hugging Face cache; subsequent runs are fast.

# 1) Interactive UI + API — record and compare live
uvicorn app:app                     # http://localhost:8000  (·/docs for the API)

# 2) Batch evaluation over the test set  (cached; safe to re-run)
python -m evaluation.run_batch      # -> results/raw_results.json

# 3) Score: WER + RTF per model
python -m evaluation.score

Tech stack

Python 3.11 · FastAPI / uvicorn · faster-whisper (CTranslate2) · google-genai · transformers + torch · jiwer · ffmpeg

Project structure

app.py                     FastAPI service + /transcribe
preprocessing.py           ffmpeg → 16 kHz mono WAV
adapters/                  one adapter per model (shared interface)
evaluation/
  run_batch.py             batch runner + caching → raw_results.json
  score.py                 normalization + WER/RTF
dataset/
  audio/                   s01..s17.m4a
  manifest.csv             filename → ground-truth reference
results/raw_results.json   raw hypotheses + timings
static/index.html          mic UI

Future work

• Domain biasing: feed the product/pesticide vocabulary as initial_prompt (Whisper) / context (Qwen) — likely to close much of the local-model gap.
• Larger local models on GPU (whisper large-v3, Qwen3-ASR 1.7B) for an accurate offline option.
• Part 2 — extraction: turn the transcript into structured care fields ({care_type, product, quantity, unit, field, date}) to pre-fill the form.