DiacriticS: Fine-Tuning an Open Language Model for Arabic Diacritisation

Authors: Youssef S. Mohamed (Mentor, KAUST), Mahdi Alkhamis (Team Leader, IAU), Mohammad Alali (KFUPM), Saad Alnafjan (KAU)

Highlights

• Contamination-controlled by construction — trained on Sadeed-Tashkeela, evaluated on SadeedDiac-25, with only 0.4% overlap between the training corpus and the Fadel test set it is scored against
• 12 open-weights models benchmarked zero-shot, scored on the mean of the Modern Standard and Classical Arabic halves of the benchmark
• LoRA under an explicit fairness contract that holds every optimisation-relevant hyperparameter identical across both 4B models — plus one full fine-tune of Qwen3.5-0.8B, so the adaptation regime itself can be compared with the base model, corpus and prompt held constant
• 4-point data-scaling curve (zero-shot, then 10/30/50%) over nested subsets — the 10% slice sits inside the 30% and that inside the 50% — so the points form a curve rather than four unrelated samples
• Corrected DER/WER scorer — NFC normalisation, letter-anchored mark comparison, reference-fixed denominator, hallucinated words counted, dagger alef included
• Bilingual project site (English / Arabic)

Introduction

Arabic Text Diacritisation (ATD) — restoring the short vowels and marks (tashkeel) that are almost always omitted from written Arabic — is a foundational preprocessing step for Text-to-Speech, Machine Translation, and Information Extraction. Without diacritics, one skeleton of letters maps to several unrelated words. The root ع ل م alone can read as:

The harder problem is measuring diacritisation honestly. Existing benchmarks suffer from data contamination — structural train/test overlap, notably between the Abbad and Fadel splits, both drawn from Tashkeela — and from register bias toward either Classical Arabic (CA) or Modern Standard Arabic (MSA). Both inflate reported accuracy.

DiacriticS builds the pipeline around that problem: a cleaned training corpus with verified minimal overlap against a balanced CA/MSA benchmark, used first to score open-weights models cold, then to fine-tune them with LoRA at several data scales.

Method

The pipeline enforces a strict wall between what the model learns from and what it is judged on:

1. Train on Sadeed-Tashkeela — the cleaned split, with the 0.4% train/test wall built in.
2. Test on SadeedDiac-25 — MSA and CA in one benchmark, 1,200 expert-reviewed paragraphs.
3. Benchmark zero-shot, across scales — score open-weights models cold, before any fine-tuning, for an honest baseline.
4. Fine-tune with LoRA, and scale the data — LoRA on a bf16 base (no QLoRA), repeated over 10%, 30% and 50% of the corpus.
5. Score, then break the score apart — DER and WER with and without sentence-final case endings, compared across CA vs. MSA to isolate where the errors come from.

DER = (Incorrect diacritics / Total characters carrying diacritics) × 100
WER = (Words with ≥1 diacritic error / Total words) × 100

Both metrics are reported with and without Case Endings (CE).

Installation

Requirements

• Python ≥ 3.9, CUDA GPU recommended (the fine-tuning runs assume 3× A100 80GB).
• A Hugging Face account and HF_TOKEN for the gated models and datasets.
• Pinned dependencies: Requirements.txt. The pins are deliberate — torch 2.10 + transformers 5.15 cannot run gemma-4-E4B-it on CUDA at all, and transformers < 5.15 does not recognise the gemma4 architecture. Each pin is annotated with the failure it prevents.

Setup

# Clone the repository
git clone https://github.com/Psycodem/DiacriticS-Language-Model-for-Arabic-Diacritisation.git
cd DiacriticS-Language-Model-for-Arabic-Diacritisation

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies (pinned — see the comments in the file)
pip install -r Requirements.txt

# Hugging Face token (needed for gated models / datasets)
export HF_TOKEN="hf_..."   # On Windows PowerShell: $env:HF_TOKEN="hf_..."

Evaluation_Functions_Corrected.py    # the scorer — every number in this repo uses it
Evaluation_Scorer_Comparison.py      # runnable diff vs the benchmark's own evaluator
Requirements.txt                     # pinned deps, annotated with what each pin prevents
Reports/
  DiacriticS_Final_Report.pdf        # the full write-up, also served from the site
Tested_Models/
  Tested Models Results.xlsx         # zero-shot results (MSA, CA, and mean)
  ibex_config/                       # the one benchmark that ran on Slurm
    Gemma_Qwen_test.py  run_gemma_qwen_test.sbatch
  modal_config/                      # the rest ran on Modal; these produced the numbers
    modal_small_models.py            # Fine-Tashkeel, Tashkeel-350M-v2
    modal_large_models.py            # aya-expanse-8b, Moonlight-16B-A3B-Instruct
    modal_fanar.py                   # Fanar-1-9B-Instruct
    modal_flan_byt5.py               # Flan-T5-Tashkeel-Small, Glonor-ByT5-Arabic
    modal_gemma3_1b.py               # gemma-3-1b-pt-10k-diacritization
    modal_qwen35_08b_plain.py        # Qwen3.5-0.8B, stock zero-shot baseline
Trained_Models/
  LoRA_Fine_Tuning_Config_Comparison.md   # the fairness contract, in full
  Gemma_4_LoRA_Diacritization/
    Train_LoRA_Gemma_4_Diacritization.py
    ibex_config/                     # 30pct / 50pct Slurm jobs
    modal_config/                    # the 10pct run
    HUGGINGFACE.md                   # where the published adapters live
    10pct/ 30pct/ 50pct/
      adapter/                       # config + model card (weights are on the Hub)
      outputs/                       # predictions, metrics, training log, loss curve
  Qwen_3_5_4B_LoRA_Diacritization/   # same shape
  Qwen_3_5_0_8B_Full_FineTune/       # full fine-tune: notebook + outputs (weights pending)
DiacriticS_Website/                  # Astro site with the live demo (deployed on Vercel)
  src/  public/  astro.config.mjs

Models on the Hub

from peft import PeftModel
model = PeftModel.from_pretrained(
    model, "Psycodem/gemma-4-e4b-lora-diacritization", subfolder="50pct")

Each subfolder carries its own model card with that run's scores and hyperparameters. The full fine-tune of Qwen3.5-0.8B is documented under Trained_Models/Qwen_3_5_0_8B_Full_FineTune/; its weights have not yet been retrieved from the training cluster.

Datasets — Misraj/Sadeed_Tashkeela for training, Misraj/SadeedDiac-25 for evaluation.

Quick Start

A) Benchmark open-weights models zero-shot

Each script loads Misraj/SadeedDiac-25 (split="train", 1,200 paragraphs), strips the diacritics to build the prompt, generates, and scores DER/WER with and without case endings.

Most of them run on Modal — one GPU, checkpointed per prediction so an interrupted run resumes rather than restarts:

modal run Tested_Models/modal_config/modal_small_models.py --probe 5 --wait   # smoke test
modal run --detach Tested_Models/modal_config/modal_small_models.py           # full 1,200

Swap in modal_large_models.py, modal_fanar.py, modal_flan_byt5.py, modal_gemma3_1b.py or modal_qwen35_08b_plain.py for the other models. Use --detach for real runs: the entrypoint spawns the job and returns, so killing the client cannot cancel the work.

On KAUST's Ibex cluster:

sbatch Tested_Models/ibex_config/run_gemma_qwen_test.sbatch

B) Score your own predictions

The corrected scorer is a drop-in replacement for the original — same function names, same return keys.

from Evaluation_Functions_Corrected import compute_der_wer

metrics = compute_der_wer(predictions, references)   # both: list[str]

print(metrics["DER_ce"], metrics["WER_ce"])      # with sentence-final case endings
print(metrics["DER_noce"], metrics["WER_noce"])  # without

C) Reproduce a LoRA fine-tuning run

Both training scripts read their configuration from environment variables. DIAC_TRAIN_FRACTION picks the data-scaling point; DIAC_RUN_TAG keeps the four runs of one model from overwriting each other's adapter and metrics.

export DIAC_TRAIN_FRACTION=0.1     # 0.1 | 0.3 | 0.5 | 1.0 — subsets are nested (seed 42)
export DIAC_RUN_TAG=10pct

torchrun --standalone --nnodes=1 --nproc_per_node=3 \
  Trained_Models/Gemma_4_LoRA_Diacritization/Train_LoRA_Gemma_4_Diacritization.py

On Ibex, the exact job files used for the published runs are checked in:

sbatch Trained_Models/Gemma_4_LoRA_Diacritization/ibex_config/run_gemma_lora_10pct.sbatch
sbatch Trained_Models/Qwen_3_5_4B_LoRA_Diacritization/ibex_config/run_qwen_lora_10pct.sbatch

Results land in eval_outputs-<tag>/metrics_summary.csv.

D) Run inference with a fine-tuned LoRA adapter

The training runs save a LoRA adapter (~165 MB), not a full model. Inference loads the original base model from the Hub and applies the adapter on top:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

BASE    = "google/gemma-4-E4B-it"        # or "Qwen/Qwen3.5-4B"
ADAPTER = "gemma-4-e4b-lora-diacritization-10pct/final_adapter"

tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

model = AutoModelForCausalLM.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)
model = PeftModel.from_pretrained(model, ADAPTER)
model = model.merge_and_unload()    # fold the adapter in — faster generation
model.eval()

merge_and_unload() is optional. Skip it to keep the adapter detachable (useful for comparing several adapters against one base); keep it for speed.

The prompt must match training exactly, or quality drops sharply for reasons that look like a bad model. Both scripts use this system prompt:

SYSTEM_PROMPT = (
    "أنت نظام متخصص في التشكيل الآلي للنصوص العربية. "
    "مهمتك إضافة الحركات (التشكيل) الصحيحة إلى النص العربي المُدخل دون تغيير الكلمات أو ترتيبها، "
    "مع مراعاة السياق النحوي والصرفي الكامل للجملة."
)

def build_messages(text):
    # Both gemma-4-E4B-it and Qwen3.5-4B accept a system role. If you swap in a
    # model whose chat template rejects one (Gemma 3 does), fold the system
    # prompt into the user turn instead:
    #     f"{SYSTEM_PROMPT}\n\n{text}"
    return [{"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user",   "content": text}]

def diacritize(text):
    msgs = build_messages(text)
    try:
        # Qwen3.5 is a reasoning model: without enable_thinking=False it emits a
        # <think> block that would be scored as if it were the diacritised text.
        prompt = tok.apply_chat_template(
            msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
    except TypeError:
        prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)

    enc = tok(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**enc, max_new_tokens=512, do_sample=False,
                             pad_token_id=tok.pad_token_id)
    gen = tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True)

    import re
    gen = re.sub(r"<think>.*?</think>", "", gen, flags=re.DOTALL)   # closed block
    if re.match(r"^\s*<think>", gen, flags=re.DOTALL):
        return ""                                                   # truncated mid-thought
    return gen.strip().strip('"“”').strip()

print(diacritize("ذهب الطالب إلى المدرسة"))

Three details that matter, each of which cost a run when it was missing:

• Greedy decoding (do_sample=False). The published numbers assume it; sampling makes results non-reproducible.
• Strip <think> blocks for Qwen. An unclosed block means generation hit max_new_tokens mid-reasoning — return an empty prediction rather than scoring reasoning text as if it were the answer.
• Left padding if you batch: tok.padding_side = "left". Right padding silently corrupts batched generation.

To reproduce the published scores rather than run ad-hoc inference, use the scoring functions in Evaluation_Functions_Corrected.py — see section B.

E) Run the project site locally

The site is an Astro app with a server-side inference proxy, so it needs Node rather than a static file server:

cd DiacriticS_Website && npm install && npm run dev

It serves on http://localhost:4321 with English at / and Arabic at /ar/. The live demo at /demo/ calls the same-origin /api/diacritize route, which forwards to a GPU endpoint; copy .env.example to .env and fill in DIACRITICS_GPU_ENDPOINT and DIACRITICS_GPU_TOKEN to enable it. Neither may carry a PUBLIC_ prefix — Astro inlines PUBLIC_ variables into the client bundle, which would publish the credential. Without them the rest of the site still runs; only the demo is inert.

Report

The full write-up — methodology, the evaluation-protocol analysis, and the qualitative error breakdown — is at Reports/DiacriticS_Final_Report.pdf, and is linked from the project site.

Performance

All numbers below are produced by Evaluation_Functions_Corrected.py, applied identically to every model and every split. Percentages; lower is better. CE = case ending (i'rab).

Zero-shot on SadeedDiac-25

Mean of the MSA and CA halves (600 paragraphs each), sorted by DER with case endings.

Per-register numbers (MSA and CA scored separately) are in Tested_Models/Tested Models Results.xlsx. The scripts that produced them are in Tested_Models/ — modal_config/ for the Modal runs, ibex_config/ for the Slurm one.

After LoRA fine-tuning

Both targets adapted under identical conditions (rank 16, alpha 32, effective batch 96, one epoch, 5% warmup ratio, seed 42) on nested subsets of Misraj/Sadeed_Tashkeela drawn with a common shuffle seed, so the 10% subset is contained in the 30% and that in the 50%. Scored on the full 1,200-paragraph SadeedDiac-25 benchmark.

Adapters, per-split predictions and training logs for every row are under Trained_Models/, and the adapters are published on the Hugging Face Hub: Psycodem/gemma-4-e4b-lora-diacritization and Psycodem/qwen3.5-4b-lora-diacritization, with each training fraction as a subfolder:

model = PeftModel.from_pretrained(
    model, "Psycodem/gemma-4-e4b-lora-diacritization", subfolder="50pct")

Key findings

1. The evaluation protocol is itself a confound. The benchmark's reference evaluator drops any paragraph in which the model altered the word count, which removes exactly the failures that separate a fluent generator from a faithful diacritiser — 15.4% of the benchmark for our Qwen3.5-4B 50% model. It also treats two Unicode orderings of shadda + vowel as different, penalising text that NFC declares identical. Scores from different harnesses cannot be compared without reconciling them first.
2. A weak starting point gains enormously from LoRA; a strong one has little room. Qwen3.5-4B falls from 58.29% to 4.38% DER, most of it purchased by the first tenth of the corpus. gemma-4-E4B-it starts at 4.18% — already below where Qwen finishes — and reaches 2.81%. Gemma leads at every fraction, but the gap narrows from 54.1 points to 1.6.
3. Sentence endings are the weak point. Every model's error rate jumps on the sentence-final, case-marking diacritic (i'rab) compared with diacritics inside a word — these models resolve local spelling far better than sentence-level syntax.
4. Classical Arabic is the harder domain. Error rates rise markedly moving from MSA passages to Classical Arabic, showing that domain-balanced training — not just more data — is what closes the gap.

References

1. Z. Aldallal, S. Chrouf, K. Hennara, M. M. Hamed, M. Hreden & S. AlModhayan, "Sadeed: Advancing Arabic Diacritization Through Small Language Model", arXiv:2504.21635, 2025.
2. A. Abbad et al., "Character-based Arabic Tashkeel Transformer (CATT)", github.com/abjadai/catt, 2023.
3. "Arabic Text Diacritization In The Age Of Transfer Learning: Token Classification Is All You Need", arXiv:2401.04848, 2024.
4. "AraXLM: Evaluating Arabic Diacritization Tools for Cross-Language Plagiarism Detection", annals-csis.org, 2024.
5. "More Data, Fewer Diacritics: Scaling Arabic TTS", 2024.
6. M. Cherradi & H. El Mahajer, "Arabic Text Diacritization Using Deep Neural Networks and Transformer-Based Architectures", Knowledge and Decision Systems with Applications, 2025.