README
scarletts-training (Tier 1)
Distillation pipeline for ScarletTTS: a compact non-autoregressive TTS student (10–30M params) trained from a corpus rendered by Kokoro-82M as the teacher. Goal: Kokoro-class naturalness at a fraction of the size and RTF, with emotion baked into a style-token table instead of runtime DSP hacks.
Architecture
phonemes ──► FFT encoder ──► duration predictor ──► length regulator ─┐
├─► conv decoder ──► iSTFT head ──► waveform
emotion id ──► style token ───────────────────────────────────────────┤
mel reference ──► GST-lite style encoder (optional) ──────────────────┘
- Teacher: Kokoro-82M (Apache-2.0) renders the training audio — no teacher weights are loaded by the student, which keeps provenance clean.
- Durations: supervised from an external aligner (whisperx timestamps or
MFA), stored per utterance in
.npz. - Loss: log-mel L1 (torchaudio transform) + duration MSE.
- Quantization: ONNX export → INT8 PTQ with corpus calibration; QAT path experimental.
Targets
| Metric | Target |
|---|---|
| Params | 10–30 M |
| File size (INT8) | ≤ 30 MB |
| RTF on Cortex-A76 (Pi 5) | ≤ 0.15 |
| Blind listening | ≥ Kokoro neutral for short assistant-style utterances |
| Emotion | 5 style tokens (neutral/happy/sad/angry/whisper) |
Pipeline
pip install -e ".[train,onnx,teacher,dev]"
# 1. Write a manifest: id,text,emotion[,voice]
# (source text: LibriTTS-R transcripts CC-BY-4.0 + assistant-domain lines)
scarletts-render-teacher --manifest data/manifest.csv --out data/rendered \
--model kokoro-v1.0.int8.onnx --voices voices-v1.0.bin
# 2. Align: extract phone-level durations into each .npz
# (whisperx word timestamps -> phone mapping; scripts/prep_align.py)
# 3. Train (GPU strongly recommended for real configs; smoke config runs on CPU)
scarletts-train-student --config configs/student_small.yaml
# Smoke test the whole loop on any CPU (~1 min):
scarletts-train-student --config configs/smoke_cpu.yaml
# 4. Export + evaluate
python -c "from slt.quantize.qat import export_onnx; ..."
scarletts-eval-rtf --onnx runs/student_small/student.onnx
Status / honesty box
| Component | State |
|---|---|
| Model code, losses, training loop | Implemented; CPU smoke test passes (tiny shapes) |
| Teacher renderer | Implemented (needs kokoro-onnx + model assets) |
| Aligner script | Stub — whisperx/MFA integration pending |
| Adversarial finetune stage | Not started (v0.2) |
| INT8 PTQ/QAT | Calibration reader ready; quantizer needs tuning to iSTFT head |
Chatterbox Nano/LoRA fine-tune track (nano_ft/) | Toolkit adopted (gokhaneraslan/chatterbox-finetuning, vendored unmodified); corpus renderer (kokoro/chatterbox, watermark-free) + prep + LoRA driver done; full-trainer smoke PASSED on turbo weights — 48 steps, loss 11.46 → 7.31 (first→last quarter mean), adapter saved; see nano_ft/notes.md |
Real training is a multi-GPU-day job; this repo ships the pipeline, configs, and gates — not pretrained weights yet.
Similar repos
Languages
Python
98.4%
Shell
1.6%
README
scarletts-training (Tier 1)
Distillation pipeline for ScarletTTS: a compact non-autoregressive TTS student (10–30M params) trained from a corpus rendered by Kokoro-82M as the teacher. Goal: Kokoro-class naturalness at a fraction of the size and RTF, with emotion baked into a style-token table instead of runtime DSP hacks.
Architecture
phonemes ──► FFT encoder ──► duration predictor ──► length regulator ─┐
├─► conv decoder ──► iSTFT head ──► waveform
emotion id ──► style token ───────────────────────────────────────────┤
mel reference ──► GST-lite style encoder (optional) ──────────────────┘
- Teacher: Kokoro-82M (Apache-2.0) renders the training audio — no teacher weights are loaded by the student, which keeps provenance clean.
- Durations: supervised from an external aligner (whisperx timestamps or
MFA), stored per utterance in
.npz. - Loss: log-mel L1 (torchaudio transform) + duration MSE.
- Quantization: ONNX export → INT8 PTQ with corpus calibration; QAT path experimental.
Targets
| Metric | Target |
|---|---|
| Params | 10–30 M |
| File size (INT8) | ≤ 30 MB |
| RTF on Cortex-A76 (Pi 5) | ≤ 0.15 |
| Blind listening | ≥ Kokoro neutral for short assistant-style utterances |
| Emotion | 5 style tokens (neutral/happy/sad/angry/whisper) |
Pipeline
pip install -e ".[train,onnx,teacher,dev]"
# 1. Write a manifest: id,text,emotion[,voice]
# (source text: LibriTTS-R transcripts CC-BY-4.0 + assistant-domain lines)
scarletts-render-teacher --manifest data/manifest.csv --out data/rendered \
--model kokoro-v1.0.int8.onnx --voices voices-v1.0.bin
# 2. Align: extract phone-level durations into each .npz
# (whisperx word timestamps -> phone mapping; scripts/prep_align.py)
# 3. Train (GPU strongly recommended for real configs; smoke config runs on CPU)
scarletts-train-student --config configs/student_small.yaml
# Smoke test the whole loop on any CPU (~1 min):
scarletts-train-student --config configs/smoke_cpu.yaml
# 4. Export + evaluate
python -c "from slt.quantize.qat import export_onnx; ..."
scarletts-eval-rtf --onnx runs/student_small/student.onnx
Status / honesty box
| Component | State |
|---|---|
| Model code, losses, training loop | Implemented; CPU smoke test passes (tiny shapes) |
| Teacher renderer | Implemented (needs kokoro-onnx + model assets) |
| Aligner script | Stub — whisperx/MFA integration pending |
| Adversarial finetune stage | Not started (v0.2) |
| INT8 PTQ/QAT | Calibration reader ready; quantizer needs tuning to iSTFT head |
Chatterbox Nano/LoRA fine-tune track (nano_ft/) | Toolkit adopted (gokhaneraslan/chatterbox-finetuning, vendored unmodified); corpus renderer (kokoro/chatterbox, watermark-free) + prep + LoRA driver done; full-trainer smoke PASSED on turbo weights — 48 steps, loss 11.46 → 7.31 (first→last quarter mean), adapter saved; see nano_ft/notes.md |
Real training is a multi-GPU-day job; this repo ships the pipeline, configs, and gates — not pretrained weights yet.
Similar repos
Languages
Python
98.4%
Shell
1.6%