MizanQA: Moroccan Legal Question Answering Benchmark

Dataset Summary

MizanQA is the first benchmark designed to evaluate large language models (LLMs) on Moroccan legal question answering tasks.
It contains 1,776 multiple-choice questions (MCQs), many with multiple correct answers, covering diverse areas of Moroccan law.
The dataset reflects the linguistic and legal complexity of Morocco, where statutes are influenced by Modern Standard Arabic, Maliki jurisprudence, customary law, and French law:contentReference[oaicite:0]{index=0}.

The benchmark was curated from publicly available Moroccan legal exam banks and MCQs. Each question was manually verified by experts to ensure accuracy.

---

Supported Tasks and Benchmarks

• Multiple-choice legal QA: Questions require choosing one or more correct answers.
• Evaluation of LLMs: MizanQA has been used to benchmark models such as LLaMA, Gemini, and Arabic-focused models like ALLAM.
• Calibration studies: Includes evaluation metrics for partial correctness and confidence calibration.

---

Dataset Structure

• Number of questions: 1,776

• Categories: 14 legal domains, including:

  • Civil Procedure (460 questions)
  • Criminal Law (847)
  • Family Code (38)
  • Family Law (66)
  • Law of Obligations and Contracts (37)
  • Judicial System of the Kingdom (88)
  • Moroccan Constitution (70)
  • Justice Sector, Exams, and others:contentReference[oaicite:1]{index=1}

• Options per question: 2–12

• Correct answers per question: 1–10

Example entry (translated to English):

{
  "question": "If it is alleged that a Pasha, first deputy to a governor... has committed a felony or misdemeanor while performing their duties, then",
  "options": [
    "The first president of the Court of Appeal ...",
    "If it is a felony, the investigating advisor ...",
    "If it is a misdemeanor, he refers the case ...",
    "Jurisdiction reverts to the Court of Cassation ...",
    "The civil party may intervene before the arbitral tribunal.",
    "The answers are correct"
  ],
  "answer": [5],
  "category": "Criminal Law"
}