Fine-tune OpenAI's Whisper for domain-specific, verbatim automatic speech recognition (ASR) using LoRA / DoRA adapters, with hyper-parameter search driven by Bayesian optimization (Optuna TPE + ASHA) on Ray Tune and a baseline-corrected WER objective. This is the code accompanying the asr4memory case study on adapting Whisper for oral history (to appear in the Zeitschrift für digitale Geisteswissenschaften, ZfdG — see Citation).
On the project's held-out oral-history test set, fine-tuning Whisper large-v3 improves word error rate from 18.6 % to 13.3 % without catastrophic forgetting on out-of-distribution audio.
A qualitative baseline-vs-fine-tuned comparison is available here.
use_dora=True) on the attention
query/key/value and output projections of Whisper.eval_wer_diff = WER(fine-tuned) − WER(pretrained) measured per validation
shard. The pretrained model is evaluated on each shard once, up front
(compute_baseline_wer), and every trial is scored against that anchor. This
isolates the adapter's effect from shard-to-shard difficulty and reduces the
objective's variance.AdapterEMACallback) and a DecorrelationStopper that kills trials whose
loss falls while WER rises (an overfitting signature).evaluation/evaluate.py streams an HDF5
test set, loads a fine-tuned adapter (or the pretrained baseline), and reports
WERasr-finetune/
├── README.md LICENSE CITATION.cff requirements.txt environment.yml
├── src/ # Python source (put src/ on PYTHONPATH)
│ ├── finetuning/ # the fine-tuning engine (compact package)
│ │ ├── train_hyper.py # ★ HPO training entry point
│ │ ├── train_single_peft.py # ★ single fixed-config run (reproduce a trial)
│ │ ├── projects_paths.py # env-driven path resolution (MODEL_PATH, DATA_PATH, ...)
│ │ ├── utils.py # shared helpers (normalize, steps_per_epoch, ...)
│ │ ├── models/ # Whisper loaders (local dir / HF hub)
│ │ ├── data_and_collator/ # HDF5 / Parquet loaders, Ray streaming collators
│ │ ├── trainers/ # training loops, PEFT build, EMA, WER metric
│ │ │ └── data/ # validation_summary_*.csv (per-shard baselines)
│ │ └── searchers_and_schedulers/# Optuna/ASHA searchers, DecorrelationStopper, spaces
│ ├── prepare_data/ # ★ materialize_dataset.py — HDF5 → Parquet features
│ └── evaluation/ # ★ evaluate.py, validate_model.py, compute_baseline_wer.py
├── configs/ # .config files (configargparse) — train/ eval/ prepare/
├── scripts/ # standalone helpers (download_hf_model.py,
│ # convert_lora_to_faster_whisper.py)
├── slurm/ # runnable SLURM job templates (+ examples/ per cluster)
└── docs/ # MONITORING.md, HPC.md, design notes
The code is run with src/ on the Python path (PYTHONPATH=src); it is not
a pip-installable package. Entry points are invoked as
PYTHONPATH=src python -m finetuning.<module> (training),
python -m evaluation.<module>, or python -m prepare_data.<module>.
git clone https://github.com/asr4memory/asr-finetune.git
cd asr-finetune
# Recommended: a fresh Python 3.12 environment (conda or venv)
conda create -n asr-finetune python=3.12 && conda activate asr-finetune
conda install -c conda-forge libsndfile # native lib for soundfile
# Install a CUDA-matched PyTorch FIRST (pick the index for your CUDA), then the rest:
pip install torch==2.5.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
environment.yml provides a one-shot conda alternative (conda env create -f environment.yml); still reinstall a CUDA-matched torch on a GPU cluster.
All machine-specific locations are read from environment variables by
src/finetuning/projects_paths.py. On a cluster, point them at fast scratch storage:
export MODEL_PATH=/path/to/models # holds <model_type>/{model,processor,tokenizer,feature_extractor}
export DATA_PATH=/path/to/data # holds the Parquet shards and *_test.h5 files
export PYTHONPATH="$PWD/src" # makes the packages importable
Optional: VALIDATION_SUMMARY_CSV (per-shard baseline CSV used by the training
objective — see step 5) and ASR_FINETUNE_ROOT (override the source root). If
unset, MODEL_PATH/DATA_PATH default to <repo>/models_local and <repo>/data.
To extend to your own cluster, copy a slurm/*.sh template and set these there.
python scripts/download_hf_model.py \
--model_id openai/whisper-large-v3 \
--output_dir "$MODEL_PATH/whisper-large-v3"
# Cache the WER metric once so later runs work offline:
python -c "import evaluate; evaluate.load('wer')"
This writes the four components (model/, processor/, feature_extractor/,
tokenizer/) into $MODEL_PATH/whisper-large-v3/, the layout the loaders expect.
--model_id nyrahealth/CrisperWhisper fetches CrisperWhisper instead.
Training reads pre-computed Parquet feature shards; the standalone evaluator
reads the HDF5 test set directly. Edit configs/prepare/materialize.config
(input hdf5_path, output_path, split, model_type, num_shards), then
materialize each split one at a time:
PYTHONPATH=src python -m prepare_data.materialize_dataset -c configs/prepare/materialize.config
# then edit split -> val_parquet (and hdf5_path -> *_val.h5) and re-run
Any config value can be overridden on the command line, e.g.
... -c configs/prepare/materialize.config --split val_parquet --hdf5_path "$DATA_PATH/..._val.h5".
Expected data directory contract: train_parquet/ and val_parquet/ shards for
training, plus <dataset_name>_test.h5 for evaluation, all under $DATA_PATH.
The HPO objective is measured relative to the pretrained model, so first record the pretrained WER on every validation shard:
PYTHONPATH=src python -m evaluation.compute_baseline_wer --model_type whisper-small
# writes src/finetuning/trainers/data/validation_summary_<tag>.csv
Point VALIDATION_SUMMARY_CSV at the CSV that matches your model and
eval_sample_fraction (e.g. validation_summary_ws_frac0.05.csv). A set of
baselines for tiny/small/medium is already committed under src/finetuning/trainers/data/.
Hyper-parameter optimization (main entry point):
PYTHONPATH=src python -m finetuning.train_hyper \
-c configs/train/small_hailmary_phase1.config \
--storage_path "$SCRATCH/ray_results" \
--optuna_db_path "$SCRATCH/optuna/small_wer_diff.db"
Key config knobs: search_schedule_mode=large_small_OPTUNA, num_samples
(trials), metric_to_optimize=eval_wer_diff, eval_sample_fraction,
hyperparameters=learning_rate_lora,warmup_ratio,alpha_coupled,target_r_wide,lora_dropout,
decorr_stopper, ema_decay. Re-running the same command resumes the Ray
Tune experiment and the Optuna study (resume_training=True).
Single fixed-config run (reproduce the best trial deterministically):
PYTHONPATH=src python -m finetuning.train_single_peft -c configs/train/small_hailmary_phase1.config
Set model_ckpt_path (the best-trial checkpoint/adapter directory) and
path_to_data in an eval config, then:
# smoke test on 5 batches first:
PYTHONPATH=src python -m evaluation.evaluate -c configs/eval/small_hailmary.config --max_eval_batches 5
# full run:
PYTHONPATH=src python -m evaluation.evaluate -c configs/eval/small_hailmary.config
The evaluator resolves nested adapter directories, guards against DoRA/peft
version mismatches, and writes per-utterance results to eval_final.json /
eval_step_<N>.json (corpus WER is the mean over utterances). It reports WER
only. Use configs/eval/<size>_baseline.config (with peft=False) for the
pretrained baseline.
To use a fine-tuned model for inference (e.g. in asr-transcribe), the PEFT
adapter has to be merged into the base model and converted to the CTranslate2
format that Faster-Whisper loads. Per job the script loads the base model, merges
the adapter, saves the merged HF model to a temp dir and runs
ct2-transformers-converter on it.
The script declares its own dependencies (PEP 723), so uv runs it standalone —
it needs neither PYTHONPATH=src nor the training environment:
# single job
uv run --script scripts/convert_lora_to_faster_whisper.py \
--adapter <best-trial checkpoint/adapter dir> \
--base openai/whisper-large-v3 \
--output <target dir>_fw
# several jobs at once
cp scripts/convert_jobs.example.toml scripts/convert_jobs.toml # then edit the paths
uv run --script scripts/convert_lora_to_faster_whisper.py --config scripts/convert_jobs.toml
scripts/convert_jobs.toml is gitignored (the paths are machine-specific).
Existing output directories are skipped unless --force is passed; quantization
defaults to float16.
For CrisperWhisper set tokenizer_override = "nyrahealth/faster_CrisperWhisper"
in the job: its HF base repo ships no tokenizer.json, and the dynamically built
fast tokenizer would assign startoftranscript the wrong id.
docs/MONITORING.md.slurm/ (fill in the <...>
placeholders); concrete worked examples for two clusters are in
slurm/examples/. See docs/HPC.md.The reported best model fine-tunes whisper-large-v3 with LoRA+DoRA on the
attention q/k/v and output projections, bf16 mixed precision, batch size 8,
random_seed = 1337. Best hyper-parameters found by the search:
| Hyper-parameter | Best value |
|---|---|
| learning rate | 1.99e-4 |
| warm-up ratio | 0.01 |
alpha multiplier (alpha_coupled) | 1 |
target rank (target_r) | 8 |
| LoRA dropout | 0.05 |
The best model came from early training (~1000 steps ≈ 8 h of audio). Training
used 4×A100 GPUs (~48 h); evaluation used a single RTX 2080 Ti (~17 h). See the
per-model baseline CSVs in src/finetuning/trainers/data/ and the design notes in
docs/design/ for the rationale behind DoRA, the α-coupling, and the resume fixes.
If you use this code, please cite the accompanying article:
Christian Horvat, Peter Kompiel, Tobias Kilgus. Adapting Automatic Speech Recognition for Oral History: A Case Study for Fine-tuning the Whisper Model with Curated and Domain-specific Training Data. Zeitschrift für digitale Geisteswissenschaften (ZfdG), forthcoming.
A machine-readable entry is in CITATION.cff.
See LICENSE.
Fine-tune OpenAI's Whisper for domain-specific, verbatim automatic speech recognition (ASR) using LoRA / DoRA adapters, with hyper-parameter search driven by Bayesian optimization (Optuna TPE + ASHA) on Ray Tune and a baseline-corrected WER objective. This is the code accompanying the asr4memory case study on adapting Whisper for oral history (to appear in the Zeitschrift für digitale Geisteswissenschaften, ZfdG — see Citation).
On the project's held-out oral-history test set, fine-tuning Whisper large-v3 improves word error rate from 18.6 % to 13.3 % without catastrophic forgetting on out-of-distribution audio.
A qualitative baseline-vs-fine-tuned comparison is available here.
use_dora=True) on the attention
query/key/value and output projections of Whisper.eval_wer_diff = WER(fine-tuned) − WER(pretrained) measured per validation
shard. The pretrained model is evaluated on each shard once, up front
(compute_baseline_wer), and every trial is scored against that anchor. This
isolates the adapter's effect from shard-to-shard difficulty and reduces the
objective's variance.AdapterEMACallback) and a DecorrelationStopper that kills trials whose
loss falls while WER rises (an overfitting signature).evaluation/evaluate.py streams an HDF5
test set, loads a fine-tuned adapter (or the pretrained baseline), and reports
WERasr-finetune/
├── README.md LICENSE CITATION.cff requirements.txt environment.yml
├── src/ # Python source (put src/ on PYTHONPATH)
│ ├── finetuning/ # the fine-tuning engine (compact package)
│ │ ├── train_hyper.py # ★ HPO training entry point
│ │ ├── train_single_peft.py # ★ single fixed-config run (reproduce a trial)
│ │ ├── projects_paths.py # env-driven path resolution (MODEL_PATH, DATA_PATH, ...)
│ │ ├── utils.py # shared helpers (normalize, steps_per_epoch, ...)
│ │ ├── models/ # Whisper loaders (local dir / HF hub)
│ │ ├── data_and_collator/ # HDF5 / Parquet loaders, Ray streaming collators
│ │ ├── trainers/ # training loops, PEFT build, EMA, WER metric
│ │ │ └── data/ # validation_summary_*.csv (per-shard baselines)
│ │ └── searchers_and_schedulers/# Optuna/ASHA searchers, DecorrelationStopper, spaces
│ ├── prepare_data/ # ★ materialize_dataset.py — HDF5 → Parquet features
│ └── evaluation/ # ★ evaluate.py, validate_model.py, compute_baseline_wer.py
├── configs/ # .config files (configargparse) — train/ eval/ prepare/
├── scripts/ # standalone helpers (download_hf_model.py,
│ # convert_lora_to_faster_whisper.py)
├── slurm/ # runnable SLURM job templates (+ examples/ per cluster)
└── docs/ # MONITORING.md, HPC.md, design notes
The code is run with src/ on the Python path (PYTHONPATH=src); it is not
a pip-installable package. Entry points are invoked as
PYTHONPATH=src python -m finetuning.<module> (training),
python -m evaluation.<module>, or python -m prepare_data.<module>.
git clone https://github.com/asr4memory/asr-finetune.git
cd asr-finetune
# Recommended: a fresh Python 3.12 environment (conda or venv)
conda create -n asr-finetune python=3.12 && conda activate asr-finetune
conda install -c conda-forge libsndfile # native lib for soundfile
# Install a CUDA-matched PyTorch FIRST (pick the index for your CUDA), then the rest:
pip install torch==2.5.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
environment.yml provides a one-shot conda alternative (conda env create -f environment.yml); still reinstall a CUDA-matched torch on a GPU cluster.
All machine-specific locations are read from environment variables by
src/finetuning/projects_paths.py. On a cluster, point them at fast scratch storage:
export MODEL_PATH=/path/to/models # holds <model_type>/{model,processor,tokenizer,feature_extractor}
export DATA_PATH=/path/to/data # holds the Parquet shards and *_test.h5 files
export PYTHONPATH="$PWD/src" # makes the packages importable
Optional: VALIDATION_SUMMARY_CSV (per-shard baseline CSV used by the training
objective — see step 5) and ASR_FINETUNE_ROOT (override the source root). If
unset, MODEL_PATH/DATA_PATH default to <repo>/models_local and <repo>/data.
To extend to your own cluster, copy a slurm/*.sh template and set these there.
python scripts/download_hf_model.py \
--model_id openai/whisper-large-v3 \
--output_dir "$MODEL_PATH/whisper-large-v3"
# Cache the WER metric once so later runs work offline:
python -c "import evaluate; evaluate.load('wer')"
This writes the four components (model/, processor/, feature_extractor/,
tokenizer/) into $MODEL_PATH/whisper-large-v3/, the layout the loaders expect.
--model_id nyrahealth/CrisperWhisper fetches CrisperWhisper instead.
Training reads pre-computed Parquet feature shards; the standalone evaluator
reads the HDF5 test set directly. Edit configs/prepare/materialize.config
(input hdf5_path, output_path, split, model_type, num_shards), then
materialize each split one at a time:
PYTHONPATH=src python -m prepare_data.materialize_dataset -c configs/prepare/materialize.config
# then edit split -> val_parquet (and hdf5_path -> *_val.h5) and re-run
Any config value can be overridden on the command line, e.g.
... -c configs/prepare/materialize.config --split val_parquet --hdf5_path "$DATA_PATH/..._val.h5".
Expected data directory contract: train_parquet/ and val_parquet/ shards for
training, plus <dataset_name>_test.h5 for evaluation, all under $DATA_PATH.
The HPO objective is measured relative to the pretrained model, so first record the pretrained WER on every validation shard:
PYTHONPATH=src python -m evaluation.compute_baseline_wer --model_type whisper-small
# writes src/finetuning/trainers/data/validation_summary_<tag>.csv
Point VALIDATION_SUMMARY_CSV at the CSV that matches your model and
eval_sample_fraction (e.g. validation_summary_ws_frac0.05.csv). A set of
baselines for tiny/small/medium is already committed under src/finetuning/trainers/data/.
Hyper-parameter optimization (main entry point):
PYTHONPATH=src python -m finetuning.train_hyper \
-c configs/train/small_hailmary_phase1.config \
--storage_path "$SCRATCH/ray_results" \
--optuna_db_path "$SCRATCH/optuna/small_wer_diff.db"
Key config knobs: search_schedule_mode=large_small_OPTUNA, num_samples
(trials), metric_to_optimize=eval_wer_diff, eval_sample_fraction,
hyperparameters=learning_rate_lora,warmup_ratio,alpha_coupled,target_r_wide,lora_dropout,
decorr_stopper, ema_decay. Re-running the same command resumes the Ray
Tune experiment and the Optuna study (resume_training=True).
Single fixed-config run (reproduce the best trial deterministically):
PYTHONPATH=src python -m finetuning.train_single_peft -c configs/train/small_hailmary_phase1.config
Set model_ckpt_path (the best-trial checkpoint/adapter directory) and
path_to_data in an eval config, then:
# smoke test on 5 batches first:
PYTHONPATH=src python -m evaluation.evaluate -c configs/eval/small_hailmary.config --max_eval_batches 5
# full run:
PYTHONPATH=src python -m evaluation.evaluate -c configs/eval/small_hailmary.config
The evaluator resolves nested adapter directories, guards against DoRA/peft
version mismatches, and writes per-utterance results to eval_final.json /
eval_step_<N>.json (corpus WER is the mean over utterances). It reports WER
only. Use configs/eval/<size>_baseline.config (with peft=False) for the
pretrained baseline.
To use a fine-tuned model for inference (e.g. in asr-transcribe), the PEFT
adapter has to be merged into the base model and converted to the CTranslate2
format that Faster-Whisper loads. Per job the script loads the base model, merges
the adapter, saves the merged HF model to a temp dir and runs
ct2-transformers-converter on it.
The script declares its own dependencies (PEP 723), so uv runs it standalone —
it needs neither PYTHONPATH=src nor the training environment:
# single job
uv run --script scripts/convert_lora_to_faster_whisper.py \
--adapter <best-trial checkpoint/adapter dir> \
--base openai/whisper-large-v3 \
--output <target dir>_fw
# several jobs at once
cp scripts/convert_jobs.example.toml scripts/convert_jobs.toml # then edit the paths
uv run --script scripts/convert_lora_to_faster_whisper.py --config scripts/convert_jobs.toml
scripts/convert_jobs.toml is gitignored (the paths are machine-specific).
Existing output directories are skipped unless --force is passed; quantization
defaults to float16.
For CrisperWhisper set tokenizer_override = "nyrahealth/faster_CrisperWhisper"
in the job: its HF base repo ships no tokenizer.json, and the dynamically built
fast tokenizer would assign startoftranscript the wrong id.
docs/MONITORING.md.slurm/ (fill in the <...>
placeholders); concrete worked examples for two clusters are in
slurm/examples/. See docs/HPC.md.The reported best model fine-tunes whisper-large-v3 with LoRA+DoRA on the
attention q/k/v and output projections, bf16 mixed precision, batch size 8,
random_seed = 1337. Best hyper-parameters found by the search:
| Hyper-parameter | Best value |
|---|---|
| learning rate | 1.99e-4 |
| warm-up ratio | 0.01 |
alpha multiplier (alpha_coupled) | 1 |
target rank (target_r) | 8 |
| LoRA dropout | 0.05 |
The best model came from early training (~1000 steps ≈ 8 h of audio). Training
used 4×A100 GPUs (~48 h); evaluation used a single RTX 2080 Ti (~17 h). See the
per-model baseline CSVs in src/finetuning/trainers/data/ and the design notes in
docs/design/ for the rationale behind DoRA, the α-coupling, and the resume fixes.
If you use this code, please cite the accompanying article:
Christian Horvat, Peter Kompiel, Tobias Kilgus. Adapting Automatic Speech Recognition for Oral History: A Case Study for Fine-tuning the Whisper Model with Curated and Domain-specific Training Data. Zeitschrift für digitale Geisteswissenschaften (ZfdG), forthcoming.
A machine-readable entry is in CITATION.cff.
See LICENSE.