nrl-ai/vn-diacritic-train — Vietnamese diacritic-restoration training data

Two register-distinct training corpora used to fine-tune Vietnamese diacritic-restoration models in the nom-vn project. Each row is a JSONL record:

{"input": "diacritic-stripped text", "target": "correctly diacriticized original"}

Inputs are produced by nom.text.strip_diacritics on the target. Both fields are NFC-normalized.

Configs

wiki_500k

500K (input, target) pairs from hirine/wikipedia-vietnamese-1M296K-dataset (CC-BY-SA-4.0). Encyclopedic register, broad topical coverage.

Filtering pipeline (deterministic, stride=7):

1. Sentence-split each article (regex on terminator + capital VN letter).
2. Drop sentences shorter than 30 chars or longer than 300.
3. Drop sentences with ASCII ratio > 95 % (URLs, tables, code blocks).
4. Drop sentences without diacritics (no training signal).
5. Drop sentences in the held-out diacritic eval set (see nrl-ai/vn-diacritic-eval).
6. Deduplicate exact target.
7. Stride-sample every 7th eligible sentence — diverse without RNG.

Includes a 5K held-out validation split (wiki_val_5k.jsonl). 0 contamination against the diacritic eval slices (audited 2026-04-30).

License: CC-BY-SA-4.0 (inherited from the source corpus).

news_150k

150K (input, target) pairs from tmnam20/Vietnamese-News-dedup (CC-BY-4.0). Modern news / business register — complements wiki_500k's encyclopedic tilt.

Same filters as wiki_500k plus:

• NFC normalization is critical here. The upstream tmnam20 dataset ships ~79 % of its sentences in NFD-decomposed form (e.g. "Cộng" stored as 'C' + 'o' + COMBINING DOT BELOW + COMBINING CIRCUMFLEX rather than the precomposed U+1ED9). Training a model on NFD targets when the eval is NFC produces silent quality regressions (we hit a -15.45 pp business-register regression before catching this). All targets in this config are NFC-normalized before write.

• Stride=3 (vs wiki's 7) since news articles are denser and shorter.

License: CC-BY-4.0 (inherited from the source corpus). More permissive than wiki_500k — derivatives don't need to be share-alike.

Loading

from datasets import load_dataset

# Wikipedia 500K
wiki = load_dataset("nrl-ai/vn-diacritic-train", "wiki_500k", split="train")
print(wiki[0])
# {'input': 'Hop dong nay duoc lap...', 'target': 'Hợp đồng này được lập...'}

# Mix wiki + news for register balance (recipe used by the published
# `nrl-ai/vn-diacritic-vit5-base` training)
import random
wiki = load_dataset("nrl-ai/vn-diacritic-train", "wiki_500k", split="train").shuffle(seed=42).select(range(350_000))
news = load_dataset("nrl-ai/vn-diacritic-train", "news_150k", split="train")
mixed = (wiki.to_list() + news.to_list())
random.Random(42).shuffle(mixed)

Eval-leak protection

Both configs are scrubbed against nrl-ai/vn-diacritic-eval (business / formal / conversational / literary slices). Audited 2026-04-30 on the released JSONL files: 0 hits across all 4 splits.

License posture

Repo-level license is CC-BY-SA-4.0 (the most restrictive of the two configs, applied for safety). Per-config licenses above. If you only need the more permissive subset, use news_150k (CC-BY-4.0).

Citation

@misc{nom_vn_diacritic_train_2026,
  title={Vietnamese diacritic-restoration training data},
  author={Nguyen, Viet-Anh and {Neural Research Lab}},
  year={2026},
  howpublished={\url{https://huggingface.co/datasets/nrl-ai/vn-diacritic-train}}
}

Cite upstream:

• hirine/wikipedia-vietnamese-1M296K-dataset
• tmnam20/Vietnamese-News-dedup