Arabic Dialect AI System — اللسان العربي

I led a 4-person team to a finalist finish in SDAIA's ALLaM Challenge (تحدي علّام — selected from 600+ participants), fine-tuning SDAIA's ALLaM LLM to bridge Modern Standard Arabic (MSA) and regional Arabic dialects: identify a dialect, translate it to MSA, and reply in-dialect.

🏆 ALLaM Challenge (تحدي علّام) · SDAIA & Tuwaiq Academy · Sep–Nov 2024 · Finalist (reached the final stage).

---

The problem it solves

Why was base ALLaM weak on dialects? A root cause in its training data. Per the ALLaM paper, its Arabic pretraining used 540B Arabic tokens — but half (270B) are machine-translated Arabic, and only 270B are natural Arabic. Translated, standard-leaning text isn't how dialects are actually spoken — so the model under-learned the rich diversity of Arabic dialects.

We probed base ALLaM across several dialects (Egyptian, Gulf, Levantine, Maghrebi): it held up only on the simplest, closest-to-MSA cases — on most dialects it misread the input. One clear example, in Moroccan Darija:

«واش كاين شي طريقة باش نبردو راسنا فهاد القايلة بلا ما نطيحو فالوهالة؟»

English: "Is there a way to cool ourselves down in this midday heat without collapsing from exhaustion?" MSA: «هل من طريقة لتبريد أنفسنا في هذا الحرّ الشديد دون أن نقع في الإرهاق؟»

— base ALLaM didn't understand the dialect and answered wrong, while a large general model handled it:

Closing that gap — teaching ALLaM to understand Arabic dialects — is the whole project. After fine-tuning, it identifies the dialect, translates it to MSA, and replies in it:

The team's chat app with my fine-tuned ALLaM integrated — identifying the dialect (“مصرية”), translating dialect → MSA, and replying in-dialect.

What it does

• Three dialect tasks in one fine-tuned model — dialect identification, dialect→MSA translation, and in-dialect response generation, trained as natural-language instructions (ما هي اللهجة / ترجم للفصحى / رد باللهجة).
• A ~1.4M-example instruction dataset built from dialect sources across ~17 Arab countries, via a full pipeline: cleaning + normalization → MinHash fuzzy deduplication → instruction formatting → balancing → size-targeted downsampling (src/data_pipeline/).
• MSA reference synthesis with a translate-then-correct LLM strategy and an experimental ROVER ensemble that fuses QARiB, MARBERT, CAMeL, GPT-3.5, and LLaMA outputs by weighted word-level voting (src/msa_generation/).
• Low-resource dialect augmentation — AraGPT2 fine-tuned to generate Najdi text where data was scarce (src/augmentation/).
• Fine-tuning + evaluation — ALLaM-13B fine-tuned on IBM watsonx (src/finetuning/), scored with BLEU / F1 / similarity plus a human-eval round (src/evaluation/), then integrated into the team's chat app.

How it works

flowchart TD
    A[Collect dialect datasets<br/>~17 Arab-country sources] --> B[Clean & normalize<br/>AraBERT + dialect char maps]
    B --> C[Fuzzy dedup<br/>MinHash + LSH]
    C --> D[Build instruction format<br/>3 tasks: identify / translate / reply]
    D --> E[Balance & downsample<br/>to target size]
    E --> F[Convert to instruction JSONL]
    F --> G[Fine-tune ALLaM-1-13B<br/>IBM watsonx.ai]
    G --> H[Evaluate<br/>BLEU · F1 · similarity · human]
    H --> I[Deploy + chat-app integration]

    K[MSA synthesis<br/>GPT-3.5 + LLaMA + ROVER] -.silver pairs.-> D
    J[AraGPT2 augmentation<br/>Najdi dialect] -.extra data.-> D

The dataset

Built from ~17 country sources and balanced across the four macro-dialects and three tasks — 1,410,265 instruction examples:

Full source catalogue: docs/dataset-catalog.md.

Results

Measured on the project's own evaluation outputs (kept under src/evaluation/sample_results/):

Honest caveat: the BLEU and F1 numbers come from two separate evaluation runs, not a single unified benchmark, and per-class F1 varied (e.g. Levantine recall was low). Some other runs failed against a since-retired endpoint. The raw outputs are included as-is rather than cherry-picked — see methodology.md.

The model was fine-tuned on IBM watsonx (Nov 2024 — loss converged, tuning complete):

The research journey

The hard part wasn't the idea — it was working out how to fine-tune ALLaM so it actually understands Arabic dialects. I researched the best approaches to Arabic dialect fine-tuning — data, methods, and evaluation — tested options, and ruled out dead-ends (encoder models can't generate; off-the-shelf classifiers only separate 1–2 dialects) before settling the design. The full story — what I read, every decision and why — is in docs/research-journey.md.

Every paper, model, dataset, and technique I drew from is credited in docs/research-references.md — annotated with what I used each for.

Tech stack

Repository structure

src/
  data_pipeline/     clean → dedup → format → balance → downsample → convert
  msa_generation/    GPT-3.5 / LLaMA translate-then-correct + ROVER ensemble
  augmentation/      AraGPT2 fine-tune for Najdi dialect (+ sample output)
  finetuning/        watsonx fine-tuning notebook + config
  inference/         ALLaM inference (Azure ML / watsonx)
  evaluation/        BLEU / F1 / similarity scripts + real sample results
docs/
  research-journey.md     how the research shaped the fine-tune (with visuals)
  research-references.md  annotated bibliography — what I read & used
  dataset-catalog.md      the dataset catalogue + composition chart
  methodology.md          the pipeline, step by step
  presentation.pdf        final challenge presentation
assets/                diagrams + demo / comparison / training screenshots

Setup & run

About model access: during the challenge, ALLaM was served on a private watsonx deployment provisioned only for participants — that endpoint is no longer available. ALLaM has since been released publicly on Hugging Face, so the model-facing code can be pointed at your own ALLaM deployment. The data-pipeline scripts run standalone on any JSON/CSV — no model needed.

# clone + install
git clone https://github.com/Omar0042/allam-arabic-dialects.git
cd allam-arabic-dialects
python -m venv .venv && . .venv/bin/activate      # (Windows: .venv\Scripts\activate)
pip install -r requirements.txt

# run a data-pipeline step on your own data — no model required
python "src/data_pipeline/fuzzy_deduplication.py"

# model-facing code (inference / evaluation) needs your own ALLaM endpoint:
cp .env.example .env        # fill in your keys/endpoints
python "src/evaluation/eval_f1.py"

Each src/data_pipeline/ script is standalone, with an if __name__ == "__main__" example you can point at your own JSON/CSV.

What's not included (and why)

This repository is a curated, partial release of a team competition project:

• No model weights or the fine-tuned ALLaM — ALLaM belongs to SDAIA and was fine-tuned/served on SDAIA's IBM watsonx environment. Only the code that builds data, fine-tunes, evaluates, and runs inference is here.
• No datasets — the training data is built from many third-party datasets, each under its own license; they are catalogued, not redistributed (see dataset-catalog.md).
• No credentials or live endpoints — all keys/tokens were removed and replaced with environment variables (.env.example).
• No teammate-owned modules — the chat-app UI and the speech-to-text component were built by teammates (I contributed the model integration), so they're excluded.
• No competition governance documents.

What's shown instead: the data + fine-tuning + evaluation + inference code I wrote, the dataset catalogue, the methodology, the research write-up, the final presentation, and real evaluation outputs.

My role

I led a 4-person team for the ~6-week challenge. I:

• scoped the team's approach within the challenge (the dialect-bridge idea and 3-task design), set the schedule, split the tasks, and ran reviews;
• did all the research myself and turned it into the design decisions (dialect scoping, dataset schema, training regime, evaluation — see research-references.md); built the dataset-research catalogue in Notion;
• drove the data pipeline (cleaning, dedup, balancing, instruction formatting); dataset collection was shared with the team;
• led fine-tuning ALLaM on IBM watsonx and ran the evaluation;
• built the inference code and integrated the fine-tuned model into the team's chat app (a teammate built the app UI).

Team & credits

• Built by a 4-person team I led: Omar Althobaiti (lead — me), Ahmad Al-Jad'ani, Ammar Abdulilah, and Omar Madani. Data collection was shared across the team. Teammates built the chat-app UI and the speech-to-text exploration (not included here — I contributed the model integration).
• ALLaM model © SDAIA. Challenge organised by SDAIA & Tuwaiq Academy.
• Open-source tools used include AraBERT, CAMeL Tools, QARiB/MARBERT, and datasketch — see dataset-catalog.md.

What I'd improve next

• One reproducible benchmark. Unify evaluation into a single script with a fixed, per-dialect held-out test set, so translation and classification are scored on the same run (today's BLEU and F1 came from separate runs).
• Real classification labels in eval. The eval scripts infer the dialect from keyword markers in the output; replace that with the model's actual predicted label (or a dedicated classifier head) for trustworthy F1.
• Quality-filter the synthesized MSA. The translate-then-correct "silver" pairs were not quality-gated; add a filter/round-trip check, and extend coverage from the 4 macro-groups toward finer dialects.

License

Source code: MIT. Does not cover the ALLaM model, third-party datasets, or team-produced presentation materials — see the note in LICENSE.