whisper-tiny-lid — 10-language, pruned, int8

Spoken language identification for de, en, es, fr, it, he, nl, pt, sv, tr, derived from openai/whisper-tiny and packaged for sherpa-onnx.

This repo was rebuilt from scratch and the previous revision used a different method. The decoder here has a 10-entry vocabulary — it is not interchangeable with a stock Whisper export. Do not mix files across revisions.

Files

No tokens.txt is needed: sherpa-onnx reads all_language_codes from the encoder metadata, and the decoder no longer has a text vocabulary.

How it was built

LID needs exactly one decoder step — feed <|startoftranscript|>, read the logits at the language-token positions. Everything else in the decoder's vocabulary machinery is dead weight, so it was removed losslessly (no retraining, no distillation):

1. Token embedding [51865, 384] → [1, 384] — only <|sot|> is ever fed.
2. Output projection [384, 51865] → [384, 10] — only the 10 language rows kept.
3. Encoder metadata rewritten so logit index i maps to LANGS[i]: n_vocab=10, sot=0, all_language_tokens=0..9, all_language_codes set to the 10 codes. Unused ids (eot, translate, transcribe, no_timestamps, no_speech, …) are clamped to 0 so nothing indexes out of bounds.

That takes the pair from ~37M to ~17M parameters. Quantization is applied after pruning: dynamic, per-channel, MatMul only (MatMulConstBOnly). Conv layers stay fp32 — dynamic int8 Conv isn't well supported and that's where most of the encoder's accuracy lives. metadata_props is re-attached after quantization, since ORT's quantizer drops it.

Usage

import sherpa_onnx

config = sherpa_onnx.SpokenLanguageIdentificationConfig(
    whisper=sherpa_onnx.SpokenLanguageIdentificationWhisperConfig(
        encoder="tiny-encoder.int8.onnx",
        decoder="tiny-decoder.int8.onnx",
    ),
    num_threads=1,
    provider="cpu",
)
slid = sherpa_onnx.SpokenLanguageIdentification(config)

s = slid.create_stream()
samples, sample_rate = sherpa_onnx.read_wave("test.wav")  # 16 kHz mono
s.accept_waveform(sample_rate, samples)
print(slid.compute(s))   # -> 'he'

Without sherpa-onnx

Plain onnxruntime works too. Run the encoder, then one decoder step with tokens=[[0]] (sot is remapped to 0) and zeroed self-attention caches; softmax over the 10 logits. See check_lid.py in the source repo.

Scope and limitations

• 10 languages only. Anything else is forced into one of the ten — there is no reject option and no "unknown" class. Softmax probabilities over 10 classes are not comparable to the original 99-way ones.
• Hebrew is he; Whisper's older exports call it iw. Both are accepted by the pruning script, the metadata emits he.
• Audio must be 16 kHz mono. Whisper pads/trims to 30 s internally; 3–10 s of speech is plenty.
• The decoder cannot transcribe. It only emits language logits.
• int8 is dynamic per-tensor/per-channel RTN; expect faster degradation than fp32 on very short (<2 s) or noisy clips.

License

MIT, following openai/whisper-tiny.