🔤 Balochi Tokenizer — Balochi_SP_47K

Production tokenizer for Southern Balochi · SentencePiece Unigram · 47,000 vocabulary · AraToken-normalized · Entropy-pruned Evaluated against 27 tokenizers across 6 research phases. Optimized for integration with the Gemma 2B Language Extension Pipeline (LEP).

---

Why This Tokenizer Exists

Standard multilingual tokenizers fragment Balochi text at 2× the rate of this custom model. BERT shatters the single word کپیٹلسٹک into 6 pieces; this tokenizer keeps it whole.

In a 4,096-token context window, this tokenizer fits ~3,593 Balochi words vs. ~1,665 for Gemma — nearly 2× more semantic content per forward pass.

---

Latest Performance Metrics (47K Vocabulary)

Quick Start

import sentencepiece as spm

sp = spm.SentencePieceProcessor()
sp.Load("balochi_sp_47k.model")

text =
 "
باسک یک لبزانکی ءُ اِلمی دیوان انت ۔ پہ زبان ءُ لبزانک ءِ دیمروئ ءَ باسکءِ تند جاہ  سال 2005 ءَ بنگیج کنگ بیتگ ۔   اے جہدءِ بندات ءَ باسک ءَ  ھالتران ، سازءُ زیمل ءِ دیمروئ، دودءُ ربیدگ ءِ پجار ءِ سرھالانی سرءَ ھم کار کُتگ ۔ اے کاروان ءَ بازینے باسک اتکگ ءُ بازینے باسک شُتگ اَنت بلے ھرکس ءَ کہ اے دیوان ءَ کلہوکے ھم کارکُتگ آئ ءِ نام چَہ ادءَ گار نہ بیتگ۔
"
tokens = sp.EncodeAsPieces(text)
ids    = sp.EncodeAsIds(text)

# Lossless roundtrip (unique to SentencePiece family)
assert sp.Decode(ids) == text

# HuggingFace-compatible wrapper
from transformers import PreTrainedTokenizerFast
tokenizer = PreTrainedTokenizerFast(tokenizer_file="balochi_sp_47k.json")

---

Performance vs. Multilingual Baselines

Evaluated on liberal capitalism.txt (5,036 words of Southern Balochi political text):

Fertility = tokens per word (lower → better). 1.14 means nearly one token per Balochi word.
UrduBERT's fertility of 7.63 confirms script similarity ≠ tokenizer compatibility.

---

Why 47K and Not 64K?

Pruning from 64K was driven by entropy analysis on the full normalized corpus (54.3M tokens):

Removing 17,000 noise tokens raises both entropy and effective vocabulary. Those tokens were diacritical variants and ZWNJ-split artifacts that only existed because the training corpus hadn't been normalized yet — they would never appear in clean text.

Cumulative frequency analysis showed that 99% of real Balochi text is covered by just 46,406 tokens. The 47K model rounds up slightly for safety.

For Gemma 2B LEP integration, this means ~17,000 fewer new embedding rows — a 26.6% reduction in embedding layer dimensions with no measurable coverage loss.

---

Normalization Pipeline

This tokenizer was trained on text pre-processed with a Balochi-adapted version of the AraToken normalization methodology:

import re, unicodedata

def normalize_balochi(text: str, drop_diacritics: bool = True,
                      preserve_ye: bool = True) -> str:
    text = unicodedata.normalize('NFKC', text)
    text = re.sub(r'[أإآٱ]', 'ا', text)
    if not preserve_ye:
        text = text.replace('ے', 'ی')
    text = text.translate(str.maketrans('٠١٢٣٤٥٦٧٨٩', '0123456789'))
    text = text.replace('؟','?').replace('؛',';').replace('،',',')
    text = text.replace('\u0640', '')
    text = text.replace('\u200C','').replace('\u200D','')
    text = text.replace('\u200F','').replace('\u061C','')
    if drop_diacritics:
        text = re.sub(r'(?<!ء)[\u064B-\u065F\u0610-\u061A\u06D6-\u06DC]', '', text)
    return re.sub(r'\s+', ' ', text).strip()

---

Training Corpus

These tokenizers were trained from scratch on a massive, deduplicated dataset comprising ~54 Million words.

The Mixed-Corpus Advantage: To prevent the models from destroying or over-fragmenting English loan-words or technical terminology, the training data strategically includes ~2 million English words seamlessly mixed with the ~52 million Balochi words. This highly optimized ratio ensures that the tokenizers can flawlessly process English text (e.g., "war industry", mathematical symbols, code) without sacrificing the vocabulary slots dedicated strictly to core Balochi morphological roots.

---

Model Variants

All SP models include byte_fallback=True — zero UNK rate on any Unicode input.

---

💻 Usage Code for Tokenizer Architectures

1. WordPiece Tokenizer (BERT-style)

from tokenizers import Tokenizer

# Load the custom WordPiece tokenizer
wp_tokenizer = Tokenizer.from_file("Balochi_WP_47K.json")

text =
 "
باسک یک لبزانکی ءُ اِلمی دیوان انت ۔ پہ زبان ءُ لبزانک ءِ دیمروئ ءَ باسکءِ تند جاہ  سال 2005 ءَ بنگیج کنگ بیتگ ۔   اے جہدءِ بندات ءَ باسک ءَ  ھالتران ، سازءُ زیمل ءِ دیمروئ، دودءُ ربیدگ ءِ پجار ءِ سرھالانی سرءَ ھم کار کُتگ ۔ اے کاروان ءَ بازینے باسک اتکگ ءُ بازینے باسک شُتگ اَنت بلے ھرکس ءَ کہ اے دیوان ءَ کلہوکے ھم کارکُتگ آئ ءِ نام چَہ ادءَ گار نہ بیتگ۔
"
encoded = wp_tokenizer.encode(text)

print("WordPiece Tokens:")
print(encoded.tokens)

2. SentencePiece Tokenizer (Gemma / LLaMA-style)

import sentencepiece as spm

# Load the custom SentencePiece model
spm_model = spm.SentencePieceProcessor()
spm_model.load("balochi_sp_47k.model")

text =
 "
باسک یک لبزانکی ءُ اِلمی دیوان انت ۔ پہ زبان ءُ لبزانک ءِ دیمروئ ءَ باسکءِ تند جاہ  سال 2005 ءَ بنگیج کنگ بیتگ ۔   اے جہدءِ بندات ءَ باسک ءَ  ھالتران ، سازءُ زیمل ءِ دیمروئ، دودءُ ربیدگ ءِ پجار ءِ سرھالانی سرءَ ھم کار کُتگ ۔ اے کاروان ءَ بازینے باسک اتکگ ءُ بازینے باسک شُتگ اَنت بلے ھرکس ءَ کہ اے دیوان ءَ کلہوکے ھم کارکُتگ آئ ءِ نام چَہ ادءَ گار نہ بیتگ۔
"
tokens = spm_model.encode_as_pieces(text)

print("SentencePiece Tokens:")
print(tokens)

3. HF BPE Tokenizer (RoBERTa / GPT-2-style)

from tokenizers import Tokenizer

# Load the custom Hugging Face BPE tokenizer
bpe_tokenizer = Tokenizer.from_file("Balochi_BPE_47K.json")

text =
 "
باسک یک لبزانکی ءُ اِلمی دیوان انت ۔ پہ زبان ءُ لبزانک ءِ دیمروئ ءَ باسکءِ تند جاہ  سال 2005 ءَ بنگیج کنگ بیتگ ۔   اے جہدءِ بندات ءَ باسک ءَ  ھالتران ، سازءُ زیمل ءِ دیمروئ، دودءُ ربیدگ ءِ پجار ءِ سرھالانی سرءَ ھم کار کُتگ ۔ اے کاروان ءَ بازینے باسک اتکگ ءُ بازینے باسک شُتگ اَنت بلے ھرکس ءَ کہ اے دیوان ءَ کلہوکے ھم کارکُتگ آئ ءِ نام چَہ ادءَ گار نہ بیتگ۔
"
encoded = bpe_tokenizer.encode(text)

print("HF BPE Tokens:")
print(encoded.tokens)

---

Training Configuration

---

Intended Uses & Limitations

Suitable for:

• Continual pre-training (CPT) of LLMs on Southern Balochi text
• BERT-style masked language modeling (use Balochi_WP_47K for WordPiece compatibility)
• Named entity recognition, text classification, sentiment analysis on Balochi
• Gemma 2B Language Extension Pipeline (LEP) via vocabulary extension + mean subtoken initialization

Limitations:

• Optimized for Southern Balochi; Northern, Eastern, and Makrani dialect performance is untested
• Evaluated on political/economic text only — cross-domain generalization (literary, social media, technical) requires validation
• No downstream task benchmarking (NER F1, MLM perplexity) has been performed yet
• English coverage is functional but not optimized; a bilingual Bal-Eng tokenizer is planned

---