nrl-ai/vn-diacritic-eval — 4-register Vietnamese diacritic-restoration eval

A reproducible evaluation set covering four registers of Vietnamese text. Used by the nom-vn project to compare diacritic-restoration models against the public Toshiiiii1/Vietnamese_diacritics_restoration_5th SOTA on a register-balanced grid.

Multi-corpus measurement is the rule — single-corpus quality numbers hide register-shift weakness. This dataset is the multi-register grid we maintain.

Splits

All sentences are NFC-normalized and stored as (input, target) pairs where input is the diacritic-stripped form and target is the correctly-diacriticized original.

Loading

from datasets import load_dataset

ds = load_dataset("nrl-ai/vn-diacritic-eval", "business_55", split="train")
print(ds[0])
# {'input': 'Hop dong so 02/HD/2025 ...', 'target': 'Hợp đồng số 02/HĐ/2025 ...'}

To evaluate a model:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from datasets import load_dataset
import unicodedata

tok = AutoTokenizer.from_pretrained("Toshiiiii1/Vietnamese_diacritics_restoration_5th")
model = AutoModelForSeq2SeqLM.from_pretrained("Toshiiiii1/Vietnamese_diacritics_restoration_5th").eval()

n_correct = n_words = 0
for split in ["business_55", "formal_72", "conversational_300", "literary_800"]:
    ds = load_dataset("nrl-ai/vn-diacritic-eval", split, split="train")
    for ex in ds:
        out = model.generate(**tok(ex["input"], return_tensors="pt"), max_length=256)
        pred = tok.decode(out[0], skip_special_tokens=True)
        # NFC + punctuation normalization both sides before comparing.
        # See nom-vn's `benchmarks/accuracy/bench_diacritic_hf_udvtb.py`.
        ...

Source attribution per split

• business_55 — benchmarks/data/diacritic_eval_v0.txt from the nom-vn repo. CC0. 55 hand-curated sentences across 4 sub-registers (15 contract, 12 official, 15 conversational, 13 news).

• formal_72 — Sentence-split from the Vietnamese translation of the Universal Declaration of Human Rights (Wikisource). Public domain. 72 sentences, formal / legal-prose register.

• conversational_300 — Sampled from Tatoeba Vietnamese sentence dump (CC-BY 2.0 FR, attribution Tatoeba and contributors). 300 conversational sentences.

• literary_800 — Extracted from the UD_Vietnamese-VTB test split (CC-BY-SA-4.0). 800 classical-literary sentences. Punctuation normalization applied to attaching characters to remove the treebank spacing convention (see nom-vn for details).

Reproducing from sources

The build scripts that produced the conversational and formal slices are in nom-vn:

• benchmarks/data/tatoeba_vi/build_diacritic_eval.py
• benchmarks/data/udhr_vi/build_diacritic_eval.py

Both deterministic (same input → same 300 / 72 sentences).

License posture

Combined dataset is CC-BY-SA-4.0 (the most restrictive of the constituent sources, applied to derivative works). Per-split licenses are noted above; if you only need a permissive subset, use business_55 (CC0) or formal_72 (public domain) which carry no share-alike obligation.

Citation

If you use this dataset, please cite both the original sources and our collation:

@misc{nom_vn_diacritic_eval_2026,
  title={4-register Vietnamese diacritic restoration evaluation},
  author={Neural Research Lab},
  year={2026},
  howpublished={\url{https://huggingface.co/datasets/nrl-ai/vn-diacritic-eval}}
}

For per-split sources (Tatoeba, UDHR, UD-VTB), follow the citation guidance on the original projects' pages.