---

Table of Contents

1. Executive Summary
2. Background & Motivation
3. Training Corpus Architecture & Statistics
4. Evaluation Methodology & Metrics
5. Phase 1 — Cross-Lingual Benchmarking (Pre-Normalization)
6. Phase 2 — Orthographic Normalization Pipeline (AraToken Integration)
7. Phase 3 — Pre- vs. Post-Normalization Comparative Analysis
8. Phase 4 — Vocabulary Size Ablation Study (12-Model Matrix)
9. Phase 5 — Information-Theoretic Analysis: Rényi Entropy
10. Phase 6 — Vocabulary Pruning: The 47K Breakthrough
11. Complete Model Registry
12. Tokenizer Selection Guide
13. Implications for Downstream NLP Tasks

• Appendix A — Training Configuration Reference
• Appendix B — Corpus Processing Time

---

1. Executive Summary

This report documents the complete end-to-end tokenizer optimization pipeline for Southern Balochi, a critically under-resourced language spoken by 10–15 million people across Balochistan (Pakistan, Iran, Afghanistan and many Gulf countries) yet nearly absent from the training data of state-of-the-art language models.

The research progressed through six distinct phases culminating in the identification of the optimal production tokenizer:

The final recommendation — Balochi_SP_47K (SentencePiece, 47,000 vocabulary) — achieves the perfect balance of linguistic coverage, embedding matrix efficiency, lossless roundtrip reconstruction, and compatibility with the downstream Gemma 2B Language Extension Pipeline (LEP). It reduces the embedding layer dimensions required for LLM fine-tuning by over 26% compared to 64K models, without sacrificing any measurable linguistic coverage.

Overall Performance Highlights

---

2. Background & Motivation

2.1 The Low-Resource Language Problem

Natural Language Processing has advanced dramatically for high-resource languages (English, French, Arabic, Chinese), but Balochi remains computationally invisible. When a Balochi speaker interacts with an AI assistant, search engine, or translation tool, the system processes their language through a tokenizer trained predominantly on other scripts and languages. The result is severe over-fragmentation: meaningful morphological units are shattered into arbitrary character sequences that carry no linguistic significance.

The tokenization penalty is not an abstract problem — it has direct consequences for context window utilization. A model with a 4,096-token context window can process approximately 2× more Balochi text per forward pass with a custom tokenizer compared to BERT or Gemma.

2.2 Why Balochi Requires Custom Tokenizers

Balochi presents four characteristics that challenge all standard multilingual tokenizers:

Modified Perso-Arabic Script: Balochi uses a specialized script incorporating characters not present in standard Arabic or Urdu, including unique diacritical markers (ءَ, ءِ, ءُ) that carry grammatical case information unique to the language.

Agglutinative Morphology: Balochi encodes grammatical relationships through suffixes and clitics that attach directly to stem forms. Preserving morphological boundaries is critical for semantic understanding in any downstream NLP task.

Data Scarcity in Multilingual Corpora: Multilingual tokenizers trained on web-crawled data encounter Balochi text extremely rarely, resulting in a vocabulary dominated by Arabic, Urdu, and Farsi subwords that poorly generalize to Balochi morphology.

Script Overlap with Semantic Divergence ("False Friends"): While Balochi shares characters with Arabic and Urdu, identical character sequences often carry entirely different meanings, making zero-shot cross-lingual transfer fundamentally unreliable.

2.3 The Fragmentation Problem Illustrated

The word کپیٹلسٹک (meaning "capitalistic") is a single morphological unit in Balochi:

BERT and Gemma split this common loanword into 6 arbitrary subword fragments — destroying the semantic unit entirely. Every additional fragment consumes precious context window capacity and forces the model to expend attention just to reassemble the original word before any linguistic processing can begin.

---

3. Training Corpus Architecture & Statistics

3.1 Corpus Components

The tokenizer training pipeline used a mixed-domain, bilingual corpus constructed by concatenating three distinct high-quality datasets. The multilingual approach ensures the tokenizer handles English loanwords and code-switching without falling back to [UNK] generation.

3.2 Corpus Preprocessing

Prior to tokenizer training, the pipeline performs the following operations:

• Validates presence of all three source files
• Merges them into a single monolithic training_corpus.txt
• Strips lines shorter than 2 characters to remove noise
• Applies NFKC normalization to unify visually identical but differently-encoded characters (e.g., standard Arabic Kaf vs. Persian Keheh)

The resulting corpus of 185,587,264 characters (~54M words) provides sufficient statistical depth for even the largest 128K vocabulary models.

3.3 Post-Normalization Corpus Statistics

After applying the AraToken-inspired normalization pipeline (Phase 2), the evaluation corpus liberal capitalism.txt changed as follows:

The full training corpus was also re-normalized, yielding training_corpus_normalized.txt (~370 MB), used for the pruning analysis.

3.4 Evaluation Text Properties

All tokenizer evaluation was conducted on two Balochi-language texts representing distinct domains:

liberal capitalism.txt — A Balochi political/economic text discussing capitalist systems and their philosophical foundations. Properties after pre-normalization:

This text is an ideal stress test because it features dense Balochi diacritical markers, mixed Balochi-English code-switching, political vocabulary with specialized terminology, and proper nouns not seen in general-purpose multilingual training data.

Balochi Labzank.txt — A Balochi political/historical text providing a complementary evaluation domain. The combination of both texts ensures evaluation metrics are not domain-biased.

---

4. Evaluation Methodology & Metrics

4.1 Primary Metrics

Each metric provides a distinct lens on tokenizer behavior:

4.2 Information-Theoretic Metrics (Phase 5)

---

5. Phase 1 — Cross-Lingual Benchmarking (Pre-Normalization)

Before constructing highly optimized models, three initial custom tokenizers (BPE-80K, WordPiece-64K, SentencePiece-64K) were trained on raw, un-normalized Balochi text and benchmarked against 14 external multilingual and language-family-specific models.

5.1 All Tokenizers Evaluated

5.2 Master Performance Results (Pre-Normalization)

The table below presents the complete tokenization metrics from the initial cross-lingual evaluation on liberal capitalism.txt:

5.3 Performance by Language Family

Aggregating by language group reveals a clear hierarchy:

5.4 Head-to-Head Group Analysis

Group 1 — WordPiece Family: Balochi WP vs. BERT Multilingual

The most critical comparison for BERT-style fine-tuning tasks:

Verdict: Balochi WordPiece wins 6 of 8 metrics. For BERT-style MLM training on Balochi text, using the custom tokenizer nearly doubles the effective context window capacity.

Group 2 — SentencePiece Family: Balochi SP vs. Gemma 2B

The most critical comparison for GPT-style/LLaMA/Gemma continual pre-training:

Verdict: Balochi SentencePiece wins 6 of 8 metrics and uniquely achieves lossless roundtrip — critical for generative language model applications. The token count reduction of 44% translates directly to processing 1.8× more Balochi text per training batch on identical hardware.

Group 3 — BPE Family: Balochi BPE vs. NLTK vs. 30K-Balochi

Verdict: NLTK leads numerically but cannot be used in neural LLMs requiring a fixed subword vocabulary. Among neural-compatible models, Balochi BPE (80K) decisively outperforms 30K-Balochi — producing 19.3% fewer tokens with zero UNK vs. 30K's catastrophic 15.54% UNK rate. The 30K tokenizer's vocabulary is insufficient for English loanwords, replacing all English text with [UNK] tokens entirely.

Group 4 — Perso-Arabic Script Family: The Urdu Lesson

Despite sharing the same Nastaliq script conventions as Balochi, UrduBERT performs catastrophically:

UrduBERT's fertility of 7.625 tokens per word — the worst of all 15 tested tokenizers — conclusively demonstrates that script similarity alone does not justify cross-lingual tokenizer transfer. Balochi and Urdu require entirely separate tokenizers.

5.5 Critical Failure Modes in Cross-Lingual Models

The run log revealed several real-world failure modes encountered during evaluation:

Context Window Overflow (Sequence Length Errors): When evaluating on the 5,036-word test corpus, several standard models triggered critical sequence-length warnings:

• BERT Multilingual: "Token indices sequence length is longer than the specified maximum sequence length for this model (9,938 > 512)" — the entire test document overflows BERT's maximum context by 19×.
• AraBERT_v2: "Token indices sequence length is longer than the specified maximum sequence length for this model (6,019 > 512)" — still 11.7× overflow.

Fatal Tokenizer Crashes:

• PersianBPETokenizer: Crashed entirely with "Exception: Unk token [UNK] not found in the vocabulary" — indicating a malformed HuggingFace tokenizer configuration.

These failures confirm that even adjacent Perso-Arabic language tokenizers are not production-safe for Balochi text.

5.6 Tokenization In Action — Worked Examples

Example 1: Single Political-Economic Term

Input: کپیٹلسٹک (capitalistic)

Example 2: Full Political Phrase

Input: لبرل کپیٹلسٹک آشوب ءِ بنیاد (Foundation of liberal capitalistic revolution)

Example 3: Author Name with Title

Input: ڈاکٹر عبدالوہاب سوری (Dr. Abdulwahab Soori)

All three custom tokenizers correctly recognize the doctor's title and full proper noun as single vocabulary entries. BERT and Gemma fragment this 3-word name into 8 and 9 arbitrary pieces respectively.

---

6. Phase 2 — Orthographic Normalization Pipeline (AraToken Integration)

6.1 Why Normalization is Critical for Balochi

Balochi web-scraped corpora contain severe orthographic noise arising from: inconsistent diacritic usage across publishers, invisible Unicode control characters from copy-pasted digital text, Hamza/Alif variants imported from Arabic typographic conventions, and both ے (Urdu Ye) and ی (Farsi Ye) being used interchangeably in different regional writing conventions.

This noise directly harms tokenizer training: identical words spelled with different invisible characters are treated as distinct surface forms, artificially inflating the long tail of the vocabulary with near-duplicates, and preventing the subword algorithms from finding true morphological boundaries.

6.2 AraToken Methodology — The Foundation

The normalization pipeline was adapted from the AraToken paper "https://arxiv.org/abs/2512.18399", which demonstrated an 8–9% fertility reduction across all algorithms for Arabic — from 1.311 to 1.199 for SentencePiece — through normalization alone, before any training changes.

The AraToken methodology maps almost perfectly onto the Balochi problem space, with critical Balochi-specific adaptations required at each step.

6.3 Complete Normalization Rules Applied

The following table details every normalization rule, its linguistic justification for Balochi, and whether it is identical to AraToken or Balochi-specific:

6.4 Python Implementation

import re
import unicodedata

def normalize_balochi(text: str, drop_diacritics: bool = True,
                       preserve_ye: bool = True) -> str:
    # Step 1: NFKC Unicode normalization
    text = unicodedata.normalize('NFKC', text)
    # Step 2: Alif variant unification
    text = re.sub(r'[أإآٱ]', 'ا', text)
    # Step 3: Ye unification (Balochi-specific — preserve by default)
    if not preserve_ye:
        text = text.replace('ے', 'ی')
    # Step 4: Arabic-Indic numeral normalization
    arabic_indic = str.maketrans('٠١٢٣٤٥٦٧٨٩', '0123456789')
    text = text.translate(arabic_indic)
    # Step 5: Arabic punctuation normalization
    text = text.replace('؟', '?').replace('؛', ';').replace('،', ',')
    # Step 6: Tatweel removal
    text = text.replace('\u0640', '')
    # Step 7: Invisible character removal (Balochi-specific)
    text = text.replace('\u200C', '').replace('\u200D', '')
    text = text.replace('\u200F', '').replace('\u061C', '')
    # Step 8: Diacritics
    if drop_diacritics:
        text = re.sub(r'(?<!ء)[\u064B-\u065F\u0610-\u061A\u06D6-\u06DC]', '', text)
    # Step 9: Whitespace normalization
    text = re.sub(r'\s+', ' ', text).strip()
    return text

6.5 Ye Preservation — The Balochi "Alif4 Equivalent"

A critical divergence from the AraToken methodology is the treatment of the ے/ی distinction. AraToken tested both "preserve" and "collapse" modes for Arabic Hamza placement, finding that preserving orthographic distinctions yielded 71% loss reduction vs. 63% for aggressive normalization in downstream LLM training.

For Balochi, the analogous distinction is the ے (Urdu Ye, U+06D2) vs. ی (Farsi Ye, U+06CC). Southern Balochi relies on this visual and grammatical distinction — largely using ے for singular/oblique and ی for other grammatical contexts. Collapsing them would destroy grammatical signal before the language model even sees the data. The decision was made to preserve this distinction in all normalized models.

---

7. Phase 3 — Pre- vs. Post-Normalization Comparative Analysis

7.1 Quantitative Impact of Normalization (64K Models, liberal capitalism.txt)

The following comparison isolates the 64K models trained on raw text (Phase 1) vs. the equivalent 64K models trained on normalized text (Phase 2), evaluated on the same normalized evaluation document:

* The Compression Ratio Paradox Explained: The mathematical compression ratio (Characters ÷ Tokens) appears lower after normalization, which seems counterintuitive. This is a mathematical artifact, not a regression. Because normalization stripped over 1,500 invisible characters and diacritics from the source text, the numerator (Characters) shrank dramatically while the denominator (Tokens) decreased by a much smaller amount. The tokenizers are performing more efficiently — proven by the universally lower Fertility scores. The denominator decrease is real and the linguistic improvement is real; only the ratio measurement is skewed by the cleaned numerator.

7.2 Fertility Improvements by Algorithm

7.3 Token Count Reductions

7.4 Continuation Rate Improvement

The ~0.5–0.6 percentage point reduction in continuation rates means ~55–136 fewer mid-word subword splits per document — directly reducing the frequency with which the model must "stitch together" word fragments to recover semantic meaning.

---

8. Phase 4 — Vocabulary Size Ablation Study (12-Model Matrix)

To overcome the limitation of selecting vocabulary sizes (64K, 80K) without empirical validation, a complete ablation study was conducted training 12 discrete tokenizer models across 3 algorithms × 4 vocabulary sizes.

8.1 Experimental Setup

8.2 Full Ablation Results (Both Texts, All 12 Models)

Balochi Labzank.txt Results

liberal capitalism.txt Results

Universal observation: All 12 models produced zero UNK tokens (UNK Rate = 0.0000%). SentencePiece is the only algorithm family achieving lossless roundtrip fidelity at every vocabulary size.

8.3 Diminishing Returns Analysis

This is the core of vocabulary ablation — identifying the inflection point where vocabulary expansion stops providing proportional compression gains.

SentencePiece — Diminishing Returns Trajectory (Averaged Across Both Texts)

The marginal return per additional 1K vocabulary tokens drops from +0.0043 (32K→64K) to +0.0017 (80K→128K) — a 60% reduction in compression return per vocabulary investment. The sharpest inflection point is clearly at 64K, where the curve breaks.

BPE — Diminishing Returns Trajectory

BPE shows the fastest rate of diminishing returns, falling to +0.0012 per +1K beyond 80K — meaning the vocabulary expansion from 80K to 128K yields only half the compression gain per vocabulary unit compared to the 32K→64K leap.

WordPiece — Diminishing Returns Trajectory

WordPiece shows an unusual pattern: the 80K→128K gain (+0.0018) slightly exceeds the 64K→80K gain (+0.0016), suggesting the intermediate 80K range is particularly inefficient for WordPiece. Despite this, the 64K→80K period still represents a dramatic slowdown from the initial 32K→64K leap.

8.4 Vocabulary Utilization — The Sparse Embedding Problem

As vocabulary size increases, the percentage of the vocabulary actually used during inference on a typical document collapses:

At 128K, single-document utilization drops to ~1.0%. This means 99% of the embedding matrix slots are not activated for any given document. In a downstream LLM, these represent dead weight embedding parameters — consuming GPU memory, initialization cost, and gradient updates for embeddings that will be rarely or never observed during training.

8.5 Vocabulary Size Recommendations from Ablation

Primary Recommendation — Maximum Context Window Efficiency: SentencePiece 128K

If the goal is packing maximum Balochi text into an LLM prompt (inference context packing), SentencePiece 128K is the definitive winner:

• Best absolute compression: 4.2571 chars/token
• Best fertility: 1.1033 tokens/word (essentially 1 token per word)
• Sustained scaling advantage: +0.0017 per +1K beyond 80K (best among all algorithms)

Secondary Recommendation — The 64K "Knee of the Curve"

For production LLM training where model memory footprint is a constraint (as larger vocabularies require a proportionally larger embedding matrix):

• The 32K→64K leap delivers the maximum return (+0.0043 compression per 1K tokens)
• After 64K, return rates drop by >50% across all algorithms
• SentencePiece 64K (Compression: 4.136, Fertility: 1.135) remains an exceptionally strong candidate

---

9. Phase 5 — Information-Theoretic Analysis: Rényi Entropy

9.1 Why Entropy Analysis?

The vocabulary ablation study showed that 128K provides the best text compression. But compression alone does not capture the full cost. Rényi Entropy analysis provides a mathematically rigorous second opinion by examining how efficiently the vocabulary is utilized from an information-theoretic perspective.

9.2 Shannon Entropy and Effective Vocabulary

Shannon Entropy (H₁) measures the uncertainty or information content of the token probability distribution:

$$H_1 = -\sum_{i} p_i \log_2 p_i$$

A lower entropy indicates a more skewed, less uniform distribution where a few tokens dominate the text.

Effective Vocabulary (V_eff) = $2^{H_1}$ — the size of a hypothetical uniform vocabulary that would produce the same entropy. It represents the number of tokens that are "doing actual work" in a document.

Vocabulary Efficiency (η) = $V_{eff} / V_{total}$ — what percentage of the actual dictionary is effectively utilized.

9.3 Entropy Trajectories Across Vocabulary Sizes

As vocabulary size increases from 32K to 128K, a critical mathematical phenomenon emerges: the Effective Vocabulary actually decreases even as the total vocabulary grows.

SentencePiece Entropy Trajectory (Balochi Labzank Text, Pre-Normalization)

WordPiece Entropy Trajectory (Tahir Hakim Text)

Post-Normalization Entropy (64K Models, liberal capitalism.txt)

9.4 The "Dead Weight" Embedding Problem Quantified

The drop in V_eff as V_total increases reveals the hidden cost of large vocabularies:

When vocabulary expands to 128K, the tokenizer learns extremely long, rare token sequences. During inference on a standard Balochi document, these rare tokens are either not used at all, or used very sparsely. The probability mass concentrates on common tokens, skewing the distribution (lowering H₁) and crushing Vocabulary Efficiency (η) to just 0.38%.

In a downstream LLM:

• A 128K vocabulary requires an embedding matrix 4× larger than 32K
• If vocabulary efficiency is only 0.38%, the vast majority of embedding parameters are under-trained
• These "dead weight" embeddings consume GPU memory during training but receive near-zero gradient updates
• They degrade model generalization by introducing noise into the parameter space

Concretely: the Effective Vocabulary for any single Balochi document is only ~482–524 tokens at the 32K level. Even at 128K, it is only ~482. The actual linguistic diversity of a Balochi document does not change with vocabulary size — only the representation format does.

9.5 Severe Long-Tail Sparsity Confirmed

The post-normalization entropy analysis reveals an extreme Zipfian distribution in Balochi text tokenization:

• Despite having 64,000 available vocabulary slots, the Effective Vocabulary per document hovers around 106–114 tokens
• The Zipf exponent is approximately s ≈ 1.05 for SentencePiece
• Min-Entropy scores (H∞) ranging from 1.80 to 2.72 indicate that the single most frequent token heavily dominates the distribution
• WordPiece suffers most severely from peak token dominance (H∞ = 1.80), meaning its most common token appears at an extremely high frequency relative to all others

9.6 The 64K Sweet Spot — Entropy Analysis Confirmation

The 64K point achieves the optimal mathematical equilibrium: strong compression (4.136) with reasonable effective vocabulary (~504 tokens). The entropy analysis confirms the ablation study finding through a completely independent mathematical lens.

SentencePiece Superiority Confirmed by Entropy: SentencePiece consistently maintains the highest Shannon Entropy and Effective Vocabulary across all sizes. At 64K, SentencePiece achieves an effective vocabulary of 503.8 tokens, while WordPiece reaches only 438.1 and BPE only 424.1. This mathematically proves that SentencePiece distributes semantic meaning more evenly across its tokens, generating a more balanced, uniform distribution for Balochi text.

---

10. Phase 6 — Vocabulary Pruning: The 47K Breakthrough

10.1 From 64K to 47K — The Pruning Rationale

Following normalization and ablation, a key insight emerged: the normalization pipeline had removed the orthographic noise that previously required rare tokens to represent. With the corpus cleaned, maintaining a 64K vocabulary meant retaining approximately 17,000 tokens that existed only to represent diacritical variants, ZWNJ-split forms, and other noise artifacts — tokens that would never appear in properly-normalized text.

The question became: what is the minimum vocabulary size needed to cover 99% of real Balochi text after normalization?

10.2 Cumulative Frequency Analysis — Terminal Output

A Vocabulary Pruning Analyzer script was executed on the full normalized training corpus (~370 MB, training_corpus_normalized.txt):

================================================================================
 ✂️ Vocabulary Pruning Analyzer — Cumulative Frequency Thresholding
==========================================================
 🎯 RECOMMENDED PRUNING THRESHOLDS
============================================================
Coverage Target      | Required Vocab Size  | Tokens Pruned
------------------------------------------------------------
90.0%                | 10,219               | 53,781
95.0%                | 21,141               | 42,859
99.0%                | 46,406               | 17,594
99.5%                | 53,240               | 10,760
99.9%                | 61,004               | 2,996
------------------------------------------------------------

10.3 Interpreting the Pruning Thresholds

The results reveal the extreme Zipfian distribution characteristic of tokenized natural language:

• 90% coverage requires only 10,219 tokens — just 16% of the 64K vocabulary
• 95% coverage requires only 21,141 tokens — 33% of the 64K vocabulary
• 99% coverage (AraToken's recommended threshold) requires 46,406 tokens — 73% of the 64K vocabulary
• The remaining 17,594 tokens (covering the bottom 1% of text) are overwhelmingly noise artifacts from the un-normalized training corpus tail

10.4 The 47K Production Model

Following the AraToken paper's 99% cumulative frequency recommendation, the decision was made to train a new production tokenizer with vocab_size = 47,000 — rounded slightly above 46,406 to provide a small safety buffer.

The Balochi_SP_47K model was then evaluated against the larger legacy models using Rényi Entropy:

This is the decisive finding: Despite removing 17,000 tokens from the 64K vocabulary, the 47K model achieves higher Shannon Entropy and utilizes a larger effective vocabulary per document than either the 64K or 128K models.

10.5 Why Pruning Improves Entropy

The 17,000 pruned tokens were "noise tokens" — rare subwords appearing fewer than ~5 times in 54 million training tokens, representing diacritical variants, ZWNJ-split artifacts, and formatting anomalies from the un-normalized corpus. These tokens had the following properties:

• Near-zero probability in the token distribution
• Pull the distribution toward extreme sparsity (very low entropy)
• Occupy embedding matrix rows that receive essentially zero gradient signal during downstream LLM training
• Crowd out semantic room in the fixed-size vocabulary

Removing them concentrates probability mass on legitimate, frequently-used morphological subwords, raising both the Shannon Entropy and the Effective Vocabulary — making the tokenizer more informationally rich per slot.

10.6 Embedding Matrix Savings for Gemma 2B

The practical benefit for the planned Language Extension Pipeline (LEP) integration with Gemma 2B is quantifiable:

A 26.6% reduction in new embedding layer dimensions directly translates to reduced GPU memory requirements for the embedding and unembedding matrices, proportionally lower initialization cost for the new Balochi token embeddings, and faster convergence during the continual pre-training phase since all 47K tokens receive adequate training signal.

---

11. Complete Model Registry

A total of 19 locally-trained models were produced across all phases:

Phase 1: Pre-Normalization Models (Original Baselines)

Phase 2: Post-Normalization Ablation Series — SentencePiece (Unigram) Family

Phase 2: Post-Normalization Ablation Series — BPE Family

Phase 2: Post-Normalization Ablation Series — WordPiece Family

---

12. Tokenizer Selection Guide

Select the appropriate tokenizer based on downstream task requirements:

---

13. Implications for Downstream NLP Tasks

13.1 Continual Pre-Training (CPT) — Compute Savings Calculation

For a training corpus of 100M Balochi character tokens:

Using the custom Balochi tokenizer reduces training token count by approximately 74% for the same textual content — directly proportional to GPU compute and time savings.

13.2 Context Window Efficiency

In a model with a 4,096-token context window, the fertility differential translates to dramatically different semantic content per forward pass:

13.3 Masked Language Model Fine-tuning

For BERT-style models fine-tuned on Balochi NLP tasks (NER, classification, sentiment):

• A 512-token BERT sequence covers ~222 Balochi words with BERT tokenizer vs. ~434 words with Balochi WordPiece
• BERT's 44 UNK tokens per ~10,000 tokens accumulate to thousands of silently-lost characters in large datasets
• Custom tokenizers eliminate truncation artifacts in documents that would otherwise overflow 512-token limits

13.4 Named Entity Recognition (NER)

Morphological boundary preservation is directly critical for NER quality. When BERT fragments a proper noun like عبدالوہاب into 5 continuation tokens, the entity boundary signal is diluted — the model must assign B-tags and I-tags across fragmented pieces rather than recognizing a single complete entity span. Custom tokenizers that preserve proper noun integrity as single tokens make NER span boundaries crisper, improving F1 scores on entity boundary detection.

13.5 Text Generation and Summarization

Lossless roundtrip fidelity (unique to SentencePiece family models) is the gold standard for generative tasks. For autoregressive generation, span extraction in QA, and copy-span tasks in summarization, the ability to map token sequences back to exact surface forms without normalization artifacts is a concrete quality advantage.

13.6 Language Extension Pipeline (LEP) for Gemma

The AraToken-demonstrated LEP approach maps directly to Balochi-Gemma integration:

AraToken demonstrated that this approach reduces evaluation loss from 8.28 to 2.43 in only 800 steps on 100K samples — under 0.01% of typical LLM pre-training budget. The 47K Balochi SP tokenizer is optimally positioned for this pipeline.

---

14. Final Conclusions & Future Work

14.1 Key Research Conclusions

Conclusion 1 — Custom Tokenizers are Mandatory

Generic multilingual tokenizers (BERT: fertility 2.305, Gemma: fertility 2.460) fragment Balochi text 2× more severely than custom tokenizers. Cross-lingual transfer from Arabic, Persian, or Urdu tokenizers fails in practice: AraBERT_v2 has a 51% UNK rate on Balochi; UrduBERT achieves a catastrophic fertility of 7.625. There is no viable alternative to training dedicated, corpus-matched tokenizers.

Conclusion 2 — Normalization is a Prerequisite, Not an Optimization

The AraToken normalization pipeline removes ~6.4% non-semantic character noise from Balochi corpora. The resulting fertility improvements (−0.026 for SentencePiece at 64K) are modest in isolation but create a prerequisite for accurate vocabulary frequency analysis and downstream pruning. Tokenizers trained on raw text embed noise patterns into the vocabulary tail; tokenizers trained on normalized text produce cleaner, more linguistically grounded subword boundaries.

Conclusion 3 — SentencePiece (Unigram) Outperforms BPE and WordPiece Across All Dimensions

SentencePiece achieves the best compression ratio at every vocabulary size, the lowest fertility at every size, the highest Shannon Entropy, the largest Effective Vocabulary, and is the only algorithm family to provide mathematically lossless roundtrip text reconstruction. It is also over 4× faster than HuggingFace's BPE implementation at encoding throughput. For all downstream Balochi NLP applications, SentencePiece is the clear algorithmic choice.

Conclusion 4 — 64K is the Compression "Knee" but 47K is the Optimal Production Vocabulary

The vocabulary ablation study shows 64K as the point where marginal compression return per additional vocabulary token drops by more than 50%. The entropy pruning analysis shows that 17,000 tokens in the 64K vocabulary are statistical noise artifacts from the un-normalized training tail. The Balochi_SP_47K model — trained after normalization, pruned at 99% cumulative frequency coverage — achieves higher entropy and larger effective vocabulary than 64K or 128K models while reducing embedding matrix dimensions by 26.6%.

Conclusion 5 — The 47K Model is Production-Ready for Gemma LEP Integration

The Balochi_SP_47K model satisfies all technical requirements for the Language Extension Pipeline:

• Zero UNK rate on any Balochi/English text (via byte_fallback)
• Lossless roundtrip fidelity (essential for generative tasks)
• 26.6% smaller embedding layer than 64K baseline
• Superior information-theoretic efficiency (higher V_eff) than all larger models
• AraToken-validated normalization pipeline ensuring clean morphological boundaries

14.2 Summary of Recommendations

14.3 Identified Limitations

1. Two-Document Evaluation Corpus: All metrics are computed on two political/economic texts. Cross-domain generalization to literary, religious, social media, or technical Balochi text requires additional evaluation.
2. Southern Dialect Focus: The corpus and evaluation texts represent Southern Balochi. Northern, Eastern, and Makrani dialect variations may show different fertility and coverage profiles.
3. No Morpheme Boundary Gold Standard: Fertility and continuation rate are proxies for morphological preservation; direct evaluation against a Balochi morpheme boundary gold standard (which does not currently exist) has not been performed.
4. Single Normalization Configuration Evaluated: The research reports results for one normalization configuration (diacritics dropped, Ye preserved). A systematic ablation across all AraToken-equivalent configuration combinations (as recommended in the Research-Paper-Suggestion) was not completed.
5. No Downstream Task Benchmarking: Tokenizer quality has not been directly validated on downstream NLP tasks (NER F1, classification accuracy, MLM perplexity) — the ultimate ground truth for tokenizer quality.

---

Appendix A — Training Configuration Reference

---

Appendix B — Corpus Processing Time

Training the 128K SentencePiece model on the full 185.5M character corpus:

all chars count = 185,587,264
✓ Trained in ~1234.5s (~20.6 minutes)
✓ Actual vocab: 128,000 (fully saturated — corpus has sufficient unique sequences)

Normalized corpus pruning analysis on ~370 MB (training_corpus_normalized.txt):

✓ Tokenization complete in 75.8s
✓ Total tokens in corpus: 54,301,475
✓ Unique tokens utilized: 63,845 (of 64,000 — 99.76% vocabulary saturation)

---