Arabic LLM Benchmarks

A comprehensive repository of Arabic LLMs benchmarks and evaluation benchmarks, curated from systematic research on evaluating Arabic Large Language Models.

📚 Survey paper: Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps

---

Table of Contents

• Overview
• Taxonomy
  1. Knowledge
     • General & STEM
     • Domain Knowledge
  2. NLP Tasks
  3. Culture & Dialects
  4. Specific-Targets
• Citation

---

📋 Overview

This survey provides the first systematic review of Arabic LLM benchmarks, analyzing 40+ evaluation benchmarks across NLP tasks, knowledge domains, cultural understanding, and specialized capabilities. We propose a taxonomy organizing benchmarks into four categories: Knowledge and STEM, NLP Tasks, Culture and Dialects, and Target-Specific evaluations. Our analysis reveals significant progress in benchmark diversity while identifying critical gaps: limited temporal evaluation, insufficient multi-turn dialogue assessment, and cultural misalignment in translated datasets. We examine three primary approaches: native collection, translation, and synthetic generation discussing their trade-offs regarding authenticity, scale, and cost. This work serves as a comprehensive reference for Arabic NLP researchers, providing insights into benchmark methodologies, reproducibility standards, and evaluation metrics while offering recommendations for future development.

---

🗂️ Taxonomy

The taxonomy formed based on existing work/benchmarks is depicted above. Each of the categories is defined below:

• Knowledge includes benchmarks evaluating acquired knowledge and reasoning capabilities, along with domain-specific benchmarks in fields such as law and medicine.
• Natural Language Processing (NLP) encompasses early task-specific benchmarks and comprehensive multi-task benchmarks, reflecting the evolution from narrow task evaluation to unified assessment across diverse dialects and domains.
• Culture and Dialects groups benchmarks assessing cultural knowledge and dialect understanding, addressing the essential property of cultural awareness in Arabic LLMs.
• Target-Specific covers benchmarks designed to assess particular LLM properties such as safety, hallucination detection, instruction-following, and vision capabilities.

---

🔬 Knowledge

⚙️ General Knowledge & STEM

🏛️ Domain Knowledge

---

💬 NLP Tasks

---

🌍 Culture & Dialects

---

🎯 Specific-Targets

---

🤝 Contributing

We welcome contributions! If you'd like to add new benchmarks or improve existing information, please feel free to submit issues or pull requests.

---

📖 Citation

If you use this repository or find the survey helpful, please cite:

@misc{alzubaidi2025evaluatingarabiclargelanguage,
      title={Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps}, 
      author={Ahmed Alzubaidi and Shaikha Alsuwaidi and Basma El Amel Boussaha and Leen AlQadi and Omar Alkaabi and Mohammed Alyafeai and Hamza Alobeidli and Hakim Hacid},
      year={2025},
      eprint={2510.13430},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.13430}, 
}