🌟 Key Features

• Includes 1,356 multimodal samples with 5,119 curated reasoning steps.
• Spans 11 diverse domains, from visual reasoning to historical and scientific analysis.
• Emphasizes step-by-step reasoning, beyond just final answer prediction.
• Each sample contains a chain of 2–6+ reasoning steps aligned to human logic.
• Curated and verified by native Arabic speakers and domain experts for linguistic and cultural fidelity.
• Built from hybrid sources: original Arabic data, high-quality translations, and synthetic samples.
• Features a robust evaluation framework measuring both final answer accuracy and reasoning quality.
• Fully open-source dataset and toolkit to support research in Arabic reasoning and multimodal AI.

🏗️ ARB Construction Pipeline

🗂️ ARB Collection

🗂️ ARB Distribution

🧪 Evaluation Protocol

🏆 Evaluation Results

• Stepwise Evaluation Using LLM-as-Judge for Closed-Source Models: | Metric ↓ / Model → | GPT-4o | GPT-4o-mini | GPT-4.1 | o4-mini | Gemini 1.5 Pro | Gemini 2.0 Flash | |----------------------------|--------|-------------|---------|---------|----------------|------------------| | Final Answer (%) | 60.22 | 52.22 | 59.43 | 58.93 | 56.70 | 57.80 | | Reasoning Steps (%) | 64.29 | 61.02 | 80.41 | 80.75| 64.34 | 64.09 |

• Stepwise Evaluation Using LLM-as-Judge for Open-Source Models: | Metric ↓ / Model → | Qwen2.5-VL | LLaMA-3.2 | AIN | LLaMA-4 Scout | Aya-Vision | InternVL3 | |----------------------------|------------|-----------|-------|----------------|-------------|-----------| | Final Answer (%) | 37.02 | 25.58 | 27.35 | 48.52 | 28.81 | 31.04 | | Reasoning Steps (%) | 64.03 | 53.20 | 52.77 | 77.70 | 63.64 | 54.50 |

📂 Dataset Structure

@misc{ghaboura2025arbcomprehensivearabicmultimodal,
      title={ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark}, 
      author={Sara Ghaboura and Ketan More and Wafa Alghallabi and Omkar Thawakar and Jorma Laaksonen and Hisham Cholakkal and Salman Khan and Rao Muhammad Anwer},
      year={2025},
      eprint={2505.17021},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2505.17021}, 
}