BAREC Shared Task 2025

Dataset Summary

BAREC (the Balanced Arabic Readability Evaluation Corpus) is a large-scale dataset developed for the BAREC Shared Task 2025, focused on fine-grained Arabic readability assessment. The dataset includes over 1M words, annotated across 19 readability levels, with additional mappings to coarser 7, 5, and 3 level schemes.

The dataset is annotated at the sentence level. Document-level readability scores are derived by assigning each document the readability level of its most difficult sentence, based on the 19-level scheme. This provides both sentence-level and document-level readability information.

---

Supported Tasks and Leaderboards

The dataset supports multi-class readability classification in the following formats:

• 19 levels (default)
• 7 levels
• 5 levels
• 3 levels

For details on the shared task, evaluation setup, and leaderboards, visit the Shared Task Website.

---

Languages

• Arabic (Modern Standard Arabic)

---

Dataset Structure

Data Instances

{'ID': 10100010008, 'Sentence': 'عيد سعيد', 'Word_Count': 2, 'Readability_Level': '2-ba', 'Readability_Level_19': 2, 'Readability_Level_7': 1, 'Readability_Level_5': 1, 'Readability_Level_3': 1, 'Annotator': 'A4', 'Document': 'BAREC_Majed_0229_1983_001.txt', 'Source': 'Majed', 'Book': 'Edition: 229', 'Author': '#', 'Domain': 'Arts & Humanities', 'Text_Class': 'Foundational'}

Data Fields

• ID: Unique sentence identifier.
• Sentence: The sentence text.
• Word_Count: Number of words in the sentence.
• Readability_Level: The readability level in 19-levels scheme, ranging from 1-alif to 19-qaf.
• Readability_Level_19: The readability level in 19-levels scheme, ranging from 1 to 19.
• Readability_Level_7: The readability level in 7-levels scheme, ranging from 1 to 7.
• Readability_Level_5: The readability level in 5-levels scheme, ranging from 1 to 5.
• Readability_Level_3: The readability level in 3-levels scheme, ranging from 1 to 3.
• Annotator: The annotator ID (A1-A5 or IAA).
• Document: Source document file name.
• Source: Document source.
• Book: Book name.
• Author: Author name.
• Domain: Domain (Arts & Humanities, STEM or Social Sciences).
• Text_Class: Readership group (Foundational, Advanced or Specialized).

Data Splits

• The BAREC dataset has three splits: Train (80%), Dev (10%), and Test (10%).
• The splits are in the document level.
• The splits are balanced accross Readability Levels, Domains, and Text Classes.

---

Evaluation

We define the Readability Assessment task as an ordinal classification task. The following metrics are used for evaluation:

• Accuracy (Acc¹⁹): The percentage of cases where reference and prediction classes match in the 19-level scheme.
• Accuracy (Acc⁷, Acc⁵, Acc³): The percentage of cases where reference and prediction classes match after collapsing the 19 levels into 7, 5, or 3 levels, respectively.
• Adjacent Accuracy (±1 Acc¹⁹): Also known as off-by-1 accuracy. The proportion of predictions that are either exactly correct or off by at most one level in the 19-level scheme.
• Average Distance (Dist): Also known as Mean Absolute Error (MAE). Measures the average absolute difference between predicted and true labels.
• Quadratic Weighted Kappa (QWK): An extension of Cohen’s Kappa that measures the agreement between predicted and true labels, applying a quadratic penalty to larger misclassifications (i.e., predictions farther from the true label are penalized more heavily).

We provide evaluation scripts here.

---

Citation

If you use BAREC in your work, please cite the following papers:

@inproceedings{elmadani-etal-2025-readability,
    title = "A Large and Balanced Corpus for Fine-grained Arabic Readability Assessment",
    author = "Elmadani, Khalid N.  and
      Habash, Nizar  and
      Taha-Thomure, Hanada",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics"
}

@inproceedings{habash-etal-2025-guidelines,
    title = "Guidelines for Fine-grained Sentence-level Arabic Readability Annotation",
    author = "Habash, Nizar  and
      Taha-Thomure, Hanada  and
      Elmadani, Khalid N.  and
      Zeino, Zeina  and
      Abushmaes, Abdallah",
    booktitle = "Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX)",
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics"
}