SadeedDiac-25: A Benchmark for Arabic Diacritization

Paper

SadeedDiac-25 is a comprehensive and linguistically diverse benchmark specifically designed for evaluating Arabic diacritization models. It unifies Modern Standard Arabic (MSA) and Classical Arabic (CA) in a single dataset, addressing key limitations in existing benchmarks.

Overview

Existing Arabic diacritization benchmarks tend to focus on either Classical Arabic (e.g., Fadel, Abbad) or Modern Standard Arabic (e.g., CATT, WikiNews), with limited domain diversity and quality inconsistencies. SadeedDiac-25 addresses these issues by:

• Combining MSA and CA in one dataset
• Covering diverse domains (e.g., news, religion, politics, sports, culinary arts)
• Ensuring high annotation quality through a multi-stage expert review process
• Avoiding contamination from large-scale pretraining corpora

Dataset Composition

SadeedDiac-25 consists of 1,200 paragraphs:

• 📘 50% Modern Standard Arabic (MSA)

  • 454 paragraphs of curated original MSA content
  • 146 paragraphs from WikiNews
  • Length: 40–50 words per paragraph

• 📗 50% Classical Arabic (CA)

  • 📖 600 paragraphs from the Fadel test set

Evaluation Results

We evaluated several models on SadeedDiac-25, including proprietary LLMs and open-source Arabic models. Evaluation metrics include Diacritic Error Rate (DER), Word Error Rate (WER), and hallucination rates. The evaluation code for this dataset is available at: https://github.com/misraj-ai/Sadeed

Evaluation Table

Note: CE = Case Ending

Citation

If you use SadeedDiac-25 in your work, please cite:

Citation

If you use this dataset, please cite:

@misc{aldallal2025sadeedadvancingarabicdiacritization,
      title={Sadeed: Advancing Arabic Diacritization Through Small Language Model}, 
      author={Zeina Aldallal and Sara Chrouf and Khalil Hennara and Mohamed Motaism Hamed and Muhammad Hreden and Safwan AlModhayan},
      year={2025},
      eprint={2504.21635},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2504.21635}, 
}

License

📄 This dataset is released under the CC BY-NC-SA 4.0 License.

Contact

📬 For questions, contact Misraj-AI on Hugging Face.