nrl-ai/vn-spell-correction-small-onnx-int8 — ONNX int8 quantization of nrl-ai/vn-spell-correction-small

Dynamic int8-quantized ONNX export of nrl-ai/vn-spell-correction-small. 75 % smaller on disk (530 MB safetensors → 307 MB ONNX int8) and no PyTorch dependency at inference time — runs on plain onnxruntime for CPU / browser / mobile deployment.

Quality on the OOD eval (n=150, hand-curated)

Same 6-slice OOD eval the source model was measured against (nrl-ai/vn-spell-correction-eval-real):

Quantization cost on aggregate: -0.25 pp word accuracy. Within the bootstrap CI overlap of the source model — no measurable quality loss.

Disk size

The fp32 ONNX export is larger than the safetensors because it unrolls the decoder twice (with-cache and without-cache paths). After int8 weight quantization, the total is comfortably under the PyTorch baseline.

Loading

from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("nrl-ai/vn-spell-correction-small-onnx-int8")
model = ORTModelForSeq2SeqLM.from_pretrained("nrl-ai/vn-spell-correction-small-onnx-int8")

inp = tok("Toi yeu Viet Nam, dat nuoc tuyet voi", return_tensors="pt")
out = model.generate(**inp, max_length=128, num_beams=1)
print(tok.decode(out[0], skip_special_tokens=True))
# "Tôi yêu Việt Nam, đất nước tuyệt vời"

pip install optimum[onnxruntime]

No PyTorch dependency required at inference time — optimum pulls onnxruntime (and transformers for the tokenizer / config).

When to use this vs the source model

• Use this when shipping to CPU-only servers, edge devices, browser (via onnxruntime-web), or mobile (onnxruntime-mobile). The 307 MB / no-PyTorch footprint matters there.
• Use nrl-ai/vn-spell-correction-small when running on GPU and PyTorch is already in the deployment. CUDA-accelerated fp16 will out-throughput int8 ONNX on a modern GPU.

Limitations

• Same training distribution as the source. All caveats from the source model card apply — in-distribution synthetic eval over-states real-world performance, Vietnamese forum slang and real Telex keystrokes are still the hardest slices.
• Dynamic int8 only. Static int8 (with calibration on a held-out set) could squeeze further size at risk of quality. Not done here because the dynamic version already meets the no-quality-loss bar.
• Beams = 1 verified. Beam search > 1 should work but isn't benched in this card.

Reproduce

git clone https://github.com/nrl-ai/nom-vn.git
cd nom-vn
pip install -e ".[diacritic-hf]"
pip install optimum[onnxruntime]

# Re-export
python training/onnx_export/export_int8.py \
    --source nrl-ai/vn-spell-correction-small \
    --output training/onnx_export/vn-spell-correction-small-onnx-int8

# Re-bench against the OOD eval
python training/onnx_export/bench_int8.py \
    --model training/onnx_export/vn-spell-correction-small-onnx-int8 \
    --json benchmarks/results/baseline_real_spell_correction_small_onnx_int8.json

License & attribution

Released under Apache 2.0 — same as the source model.

@misc{nom_vn_spell_correction_onnx_int8_2026,
  title={Vietnamese Spell Correction — ONNX int8 quantization for edge deployment},
  author={Nguyen, Viet-Anh and {Neural Research Lab}},
  year={2026},
  howpublished={\url{https://huggingface.co/nrl-ai/vn-spell-correction-small-onnx-int8}}
}

See also

• Source model: nrl-ai/vn-spell-correction-small
• Toolkit repo: https://github.com/nrl-ai/nom-vn
• Eval set: nrl-ai/vn-spell-correction-eval-real