An advanced generative AI framework by NVIDIA designed for creating and deploying large-scale language and multimodal models, with extensive support for ASR, TTS, and computer vision.
Python
1
9,019 commits
updated Oct 5, 2026
[!IMPORTANT] Want faster improvements? This project is entirely self-funded on my minimum-wage salary, and every training run competes for a single consumer GPU. If you or your organisation can donate an RTX 5090, or the money to buy one second-hand, it would directly speed up the next versions of the dataset and the fine-tuned models. Reach out via olicorne.org or open an issue on this repository.
[!WARNING] This documentation was written by an AI agent (Claude Code) and very likely contains some degree of hallucination. Treat it as a guided tour, not as a specification: the code, the configs and the commit history are the source of truth, and where they disagree with a sentence below, they win.
It is written this way on purpose. I published this to be useful to other people, and I cannot afford the time to document all of it properly by hand. The expected way to onboard is to point an LLM at the repository and have it check every claim against the actual files, which is also how it was written. If you find something wrong, an issue is welcome.
This repository is a fork of NVIDIA-NeMo/NeMo. It is not a general-purpose NeMo distribution and it is not maintained as one. It exists to document, reproducibly, how Olicorne/parakeet-tdt-0.6b-v3-UltiMed-onnx was fine-tuned: the patches NeMo needed, the training configuration, and the reasoning behind both. If you want NeMo itself, go upstream.
Everything here sits on top of upstream commit
6442018984
([TTS][MagpieTTS] Longform TTS using MagpieTTS (#15210), 2025-12-23). That is the exact tree
the released model was trained on. The branch has deliberately not been rebased onto a newer
NeMo: the patches below were never tested against anything else, and the point of this repo is to
record what actually ran.
This fork is one piece of a French medical ASR stack; every other piece is public.
| Repository | What it is |
|---|---|
| Olicorne/UltiMed-ASR-FR-v1 | The dataset the fine-tune was trained on: 601,338 clips / 3,105 h of synthesized French medical speech, plus an eval-only PARROT subset. |
| Olicorne/parakeet-tdt-0.6b-v3-UltiMed-onnx | The French medical fine-tune trained on that dataset, exported to ONNX (fp32 / fp16 / int8 / w4a8). |
| Olicorne/parakeet-tdt-0.6b-v3-optimized-onnx | The multilingual baseline the fine-tune builds on: the upstream ONNX re-quantized for int8 accuracy on long audio and graph-optimized for browser speed. |
| UltiMed-ASR-FR-v1-scripts | The open recipe that built that dataset end to end: text sources, spoken-form normalization, the batch synthesis client, and the transcribe-and-rescore QC pass. |
| This repository | The NeMo fork and training configs used for the fine-tune, documented here. |
| UltiMed-ASR-FR-v1-Voxtral | The Voxtral TTS container that spoke every clip of the dataset: Dockerfile, tuning, and the vllm-omni patches. |
| Parakeet Web | The in-browser ASR app that loads either ONNX model, live at parakeetweb.olicorne.org. Everything runs client-side. |
Upstream base model: nvidia/parakeet-tdt-0.6b-v3, on top of NVIDIA-NeMo/NeMo.
The detailed engineering notes are in the tree itself, not in a separate document, so they
cannot drift away from what they describe. perso/training_config.yaml is annotated at length:
the VRAM cost model and the ten measured OOM points it was fitted on, why each setting is the
value it is, and what to change first when it has to move. The commit messages below explain
each patch. The launcher scripts document themselves in their headers.
The history is squashed into one commit per coherent change, oldest first. Individual commits are not each independently runnable (the config that ties everything together arrives last); only the branch tip is a working tree. Each commit message explains the why at length.
asr: batch by duration cost instead of clip count
The RNNT/TDT joint builds a dense [B, T, U, V] lattice, so a clip's memory cost grows with the
square of its duration and a fixed batch size has to be sized for the worst clip in the
manifest. DurationCostBatchSampler packs each batch to a memory budget instead. Fitted against
ten measured OOM points on a 24 GB card. Net effect on the real corpus: 1.56x throughput and
96.6% of corpus hours kept instead of 86.8%.
optim: add AdamSPD (Selective Projection Decay)
AdamW decays toward zero, which is the wrong direction when the goal is to not forget the
pretrained multilingual model. SPD decays toward the pretrained weights, selectively. Includes
the sign fix for a bug where two errors cancelled and no decay was ever applied, so every
weight_decay value produced bit-identical runs.
metrics: normalised WER and per-source macro averaging
Score after normalising casing, punctuation and number/unit spelling, and weight each validation
source equally, so a 30-minute real-speech set counts as much as a multi-hour synthetic one.
finetune: cached-encoder training path and latent augmentation
The training entry point (examples/asr/speech_to_text_finetune_cached.py, used for both modes),
plus an optional path that freezes the lower encoder layers and precomputes their output. Because
a frozen prefix means waveform augmentors no longer reach the trainable part, augmentation is
applied in latent space instead.
config: audit that every setting is actually read, and crash if not
Six settings in the config turned out to do nothing, and all six looked correct in the file. Two
layers: reject any key nothing read, then re-read the settings that matter back off the objects
that were actually built. Runs before the first training step, because these are expensive to
discover on day five of a six-day run.
export: opt-in web-export encoder flags and ONNX exporter
Off-by-default encoder export flags that make the exported encoder usable under onnxruntime-web, plus the
exporter and the in-domain int8 calibration set. Originally a mask-free graph with a padded-batch NaN tripwire
(batch 1 or equal-length batches only); since 2026-10-02 a key-only additive padding bias instead
(export_key_pad_mask: one Add per layer, bit-identical at batch 1, correct for mixed-length batches), plus the
runtime relative-position encoding. Changed with Claude Code.
data: corpus preparation, rehearsal sets and the manifests
Everything that builds the manifests: the LLM-generated in-domain sentence sets, the FLEURS and
Common Voice rehearsal material used to measure and limit forgetting, and the UltiMed validation
slices.
train: launcher, training config and notes (this commit)
training_start.sh, tensorboard_start.sh, the full annotated perso/training_config.yaml, and
the engineering notes.
git clone https://github.com/thiswillbeyourgithub/UltiMed-ASR-FR-v1-NeMo_training_scripts
uv venv --python=3.10.12
uv pip install nemo_toolkit[asr] onnx==1.18.0 numba-cuda==0.15.1 cuda-core cuda-bindings \
cuda-pathfinder cuda-python==13.1.1 tensorboard>=2.19.0 setuptools==81.0.0 \
--force-reinstall loguru deepspeed
cp .env.example .env # set NEMO_EXP_DIR to a path on a drive with ~22 GB free per run
./training_start.sh
uv.lock pins the exact resolved environment, and perso/obsolete/working_pip_list is the
uv pip list of the venv that actually produced the released model. Keep the onnx<1.19 cap if
you re-lock: 1.19 hard-imports ml_dtypes.float4_e2m1fn, which the numpy<2 pin makes
unsatisfiable, and NeMo then fails to import at all.
Trained on a single RTX 3090 Ti (24 GB). The VRAM budget in the config is fitted to that card;
read the comments around cost_batching in perso/training_config.yaml before changing
max_duration or cost_batching.budget, which are coupled, and re-run
NEMO_EXP_DIR=/tmp/oom_check ./perso/oom_margin_check.sh afterwards.
Validation was checked separately rather than argued about, because compute_eval_loss defaults
to true and validation therefore builds the same joint lattice training does. Nine of the fifteen
validation sets score above the 15000 training budget on the cost model (worst: fleurs_it at
37344), and num_sanity_val_steps cannot catch a problem there because validation sets have
shuffle: false, so the sanity pass sees the first batches rather than the longest ones. A
validation OOM would land about 3.5 h in, before the first checkpoint exists. Measured instead
(2026-08-22, with Claude Code) on a fixture holding the two longest batches of every one of the
15 validation sets, run after one training batch so the optimizer state is resident: exit 0,
peak 21363 of 24564 MiB, no OOM, all 15 dataloaders completed. Validation is the cheaper phase
despite the longer clips. Re-run that check if the validation batch sizes or manifests change.
Four entries in perso/ look like duplicates of files that also live in the normal NeMo layout:
cached_encoder_dataset.py, latent_augment.py, speech_to_text_finetune.py and
speech_to_text_finetune_cached.py (the last being the training entry point). They are symlinks
into the live tree, not copies, so there is only ever one real file to edit. They started out as
genuine byte-identical copies and were converted once that was noticed.
perso/oli_spoken_dataset/, 205 clips of my own voice reading sentences containing drug names aloud. It was the
only real (non-synthetic) in-domain signal in the whole setup and one of the six equally weighted
checkpoint monitors, so it is still referenced in training_config.yaml to keep that an accurate
record. It will not resolve on a fresh clone. See the note at those entries for how to run
without it..wav files behind perso/nemo_dataset/ and perso/drug_sentence_dataset/
are gitignored; only the manifests are here. Regenerate them with the TTS container linked above.training_start.sh used to free GPU memory before a
run. It hardcoded paths on my machine.perso/*_ignore-backups). Paths in the config are relative to those symlinks, so nothing
machine-specific is committed.Upstream NeMo is Apache-2.0 and remains so; see LICENSE. The patches and the perso/
scripts in this fork are offered under the same licence.
Note that the PARROT subset of the training corpus is CC BY-NC-SA and was used evaluation-only, never for training.
Much of the work in this fork was done with AI coding agents, principally
Claude Code and, for a handful of commits, aider.
The Co-authored-by trailers in the original unsquashed history recorded which model made which
change; that detail is lost in the squash, so it is stated here instead. The measurements,
diagnoses and design decisions were reviewed and run by me.
Upstream NeMo's own README is available in the upstream repository.
An advanced generative AI framework by NVIDIA designed for creating and deploying large-scale language and multimodal models, with extensive support for ASR, TTS, and computer vision.
Python
1
9,019 commits
updated Oct 5, 2026
[!IMPORTANT] Want faster improvements? This project is entirely self-funded on my minimum-wage salary, and every training run competes for a single consumer GPU. If you or your organisation can donate an RTX 5090, or the money to buy one second-hand, it would directly speed up the next versions of the dataset and the fine-tuned models. Reach out via olicorne.org or open an issue on this repository.
[!WARNING] This documentation was written by an AI agent (Claude Code) and very likely contains some degree of hallucination. Treat it as a guided tour, not as a specification: the code, the configs and the commit history are the source of truth, and where they disagree with a sentence below, they win.
It is written this way on purpose. I published this to be useful to other people, and I cannot afford the time to document all of it properly by hand. The expected way to onboard is to point an LLM at the repository and have it check every claim against the actual files, which is also how it was written. If you find something wrong, an issue is welcome.
This repository is a fork of NVIDIA-NeMo/NeMo. It is not a general-purpose NeMo distribution and it is not maintained as one. It exists to document, reproducibly, how Olicorne/parakeet-tdt-0.6b-v3-UltiMed-onnx was fine-tuned: the patches NeMo needed, the training configuration, and the reasoning behind both. If you want NeMo itself, go upstream.
Everything here sits on top of upstream commit
6442018984
([TTS][MagpieTTS] Longform TTS using MagpieTTS (#15210), 2025-12-23). That is the exact tree
the released model was trained on. The branch has deliberately not been rebased onto a newer
NeMo: the patches below were never tested against anything else, and the point of this repo is to
record what actually ran.
This fork is one piece of a French medical ASR stack; every other piece is public.
| Repository | What it is |
|---|---|
| Olicorne/UltiMed-ASR-FR-v1 | The dataset the fine-tune was trained on: 601,338 clips / 3,105 h of synthesized French medical speech, plus an eval-only PARROT subset. |
| Olicorne/parakeet-tdt-0.6b-v3-UltiMed-onnx | The French medical fine-tune trained on that dataset, exported to ONNX (fp32 / fp16 / int8 / w4a8). |
| Olicorne/parakeet-tdt-0.6b-v3-optimized-onnx | The multilingual baseline the fine-tune builds on: the upstream ONNX re-quantized for int8 accuracy on long audio and graph-optimized for browser speed. |
| UltiMed-ASR-FR-v1-scripts | The open recipe that built that dataset end to end: text sources, spoken-form normalization, the batch synthesis client, and the transcribe-and-rescore QC pass. |
| This repository | The NeMo fork and training configs used for the fine-tune, documented here. |
| UltiMed-ASR-FR-v1-Voxtral | The Voxtral TTS container that spoke every clip of the dataset: Dockerfile, tuning, and the vllm-omni patches. |
| Parakeet Web | The in-browser ASR app that loads either ONNX model, live at parakeetweb.olicorne.org. Everything runs client-side. |
Upstream base model: nvidia/parakeet-tdt-0.6b-v3, on top of NVIDIA-NeMo/NeMo.
The detailed engineering notes are in the tree itself, not in a separate document, so they
cannot drift away from what they describe. perso/training_config.yaml is annotated at length:
the VRAM cost model and the ten measured OOM points it was fitted on, why each setting is the
value it is, and what to change first when it has to move. The commit messages below explain
each patch. The launcher scripts document themselves in their headers.
The history is squashed into one commit per coherent change, oldest first. Individual commits are not each independently runnable (the config that ties everything together arrives last); only the branch tip is a working tree. Each commit message explains the why at length.
asr: batch by duration cost instead of clip count
The RNNT/TDT joint builds a dense [B, T, U, V] lattice, so a clip's memory cost grows with the
square of its duration and a fixed batch size has to be sized for the worst clip in the
manifest. DurationCostBatchSampler packs each batch to a memory budget instead. Fitted against
ten measured OOM points on a 24 GB card. Net effect on the real corpus: 1.56x throughput and
96.6% of corpus hours kept instead of 86.8%.
optim: add AdamSPD (Selective Projection Decay)
AdamW decays toward zero, which is the wrong direction when the goal is to not forget the
pretrained multilingual model. SPD decays toward the pretrained weights, selectively. Includes
the sign fix for a bug where two errors cancelled and no decay was ever applied, so every
weight_decay value produced bit-identical runs.
metrics: normalised WER and per-source macro averaging
Score after normalising casing, punctuation and number/unit spelling, and weight each validation
source equally, so a 30-minute real-speech set counts as much as a multi-hour synthetic one.
finetune: cached-encoder training path and latent augmentation
The training entry point (examples/asr/speech_to_text_finetune_cached.py, used for both modes),
plus an optional path that freezes the lower encoder layers and precomputes their output. Because
a frozen prefix means waveform augmentors no longer reach the trainable part, augmentation is
applied in latent space instead.
config: audit that every setting is actually read, and crash if not
Six settings in the config turned out to do nothing, and all six looked correct in the file. Two
layers: reject any key nothing read, then re-read the settings that matter back off the objects
that were actually built. Runs before the first training step, because these are expensive to
discover on day five of a six-day run.
export: opt-in web-export encoder flags and ONNX exporter
Off-by-default encoder export flags that make the exported encoder usable under onnxruntime-web, plus the
exporter and the in-domain int8 calibration set. Originally a mask-free graph with a padded-batch NaN tripwire
(batch 1 or equal-length batches only); since 2026-10-02 a key-only additive padding bias instead
(export_key_pad_mask: one Add per layer, bit-identical at batch 1, correct for mixed-length batches), plus the
runtime relative-position encoding. Changed with Claude Code.
data: corpus preparation, rehearsal sets and the manifests
Everything that builds the manifests: the LLM-generated in-domain sentence sets, the FLEURS and
Common Voice rehearsal material used to measure and limit forgetting, and the UltiMed validation
slices.
train: launcher, training config and notes (this commit)
training_start.sh, tensorboard_start.sh, the full annotated perso/training_config.yaml, and
the engineering notes.
git clone https://github.com/thiswillbeyourgithub/UltiMed-ASR-FR-v1-NeMo_training_scripts
uv venv --python=3.10.12
uv pip install nemo_toolkit[asr] onnx==1.18.0 numba-cuda==0.15.1 cuda-core cuda-bindings \
cuda-pathfinder cuda-python==13.1.1 tensorboard>=2.19.0 setuptools==81.0.0 \
--force-reinstall loguru deepspeed
cp .env.example .env # set NEMO_EXP_DIR to a path on a drive with ~22 GB free per run
./training_start.sh
uv.lock pins the exact resolved environment, and perso/obsolete/working_pip_list is the
uv pip list of the venv that actually produced the released model. Keep the onnx<1.19 cap if
you re-lock: 1.19 hard-imports ml_dtypes.float4_e2m1fn, which the numpy<2 pin makes
unsatisfiable, and NeMo then fails to import at all.
Trained on a single RTX 3090 Ti (24 GB). The VRAM budget in the config is fitted to that card;
read the comments around cost_batching in perso/training_config.yaml before changing
max_duration or cost_batching.budget, which are coupled, and re-run
NEMO_EXP_DIR=/tmp/oom_check ./perso/oom_margin_check.sh afterwards.
Validation was checked separately rather than argued about, because compute_eval_loss defaults
to true and validation therefore builds the same joint lattice training does. Nine of the fifteen
validation sets score above the 15000 training budget on the cost model (worst: fleurs_it at
37344), and num_sanity_val_steps cannot catch a problem there because validation sets have
shuffle: false, so the sanity pass sees the first batches rather than the longest ones. A
validation OOM would land about 3.5 h in, before the first checkpoint exists. Measured instead
(2026-08-22, with Claude Code) on a fixture holding the two longest batches of every one of the
15 validation sets, run after one training batch so the optimizer state is resident: exit 0,
peak 21363 of 24564 MiB, no OOM, all 15 dataloaders completed. Validation is the cheaper phase
despite the longer clips. Re-run that check if the validation batch sizes or manifests change.
Four entries in perso/ look like duplicates of files that also live in the normal NeMo layout:
cached_encoder_dataset.py, latent_augment.py, speech_to_text_finetune.py and
speech_to_text_finetune_cached.py (the last being the training entry point). They are symlinks
into the live tree, not copies, so there is only ever one real file to edit. They started out as
genuine byte-identical copies and were converted once that was noticed.
perso/oli_spoken_dataset/, 205 clips of my own voice reading sentences containing drug names aloud. It was the
only real (non-synthetic) in-domain signal in the whole setup and one of the six equally weighted
checkpoint monitors, so it is still referenced in training_config.yaml to keep that an accurate
record. It will not resolve on a fresh clone. See the note at those entries for how to run
without it..wav files behind perso/nemo_dataset/ and perso/drug_sentence_dataset/
are gitignored; only the manifests are here. Regenerate them with the TTS container linked above.training_start.sh used to free GPU memory before a
run. It hardcoded paths on my machine.perso/*_ignore-backups). Paths in the config are relative to those symlinks, so nothing
machine-specific is committed.Upstream NeMo is Apache-2.0 and remains so; see LICENSE. The patches and the perso/
scripts in this fork are offered under the same licence.
Note that the PARROT subset of the training corpus is CC BY-NC-SA and was used evaluation-only, never for training.
Much of the work in this fork was done with AI coding agents, principally
Claude Code and, for a handful of commits, aider.
The Co-authored-by trailers in the original unsquashed history recorded which model made which
change; that detail is lost in the squash, so it is stated here instead. The measurements,
diagnoses and design decisions were reviewed and run by me.
Upstream NeMo's own README is available in the upstream repository.