Sakinah-AI at MentalQA: A Comparative Study of Few-Shot, Optimized, and Ensemble Methods for Arabic Mental Health Question Classification

Fatimah Emad Elden, Mumina Abukar

Cairo University & The University of South Wales

---

🏆 Overview

This repository contains the code and resources for our submission to the MentalQA 2025 Shared Task, Track 1. Our work focuses on the multi-label classification of user questions in Arabic within the mental health domain. We conduct a comprehensive comparative analysis of few-shot learning with Large Language Models (LLMs) against several fine-tuning strategies using two base models: CAMeL-BERT and AraBERT. For each base model, we evaluate a single, hyperparameter-optimized model against a 5-fold cross-validation ensemble. Our findings reveal that few-shot learning with a large, specialized LLM and a single, optimized CAMeL-BERT model are the most effective strategies for this task.

---

🔑 Key Findings

• Few-Shot Learning Excels: Few-shot learning with Writer/Palmyra-Med-70B-32K achieved the highest weighted F1-score of 0.605.
• Optimized Fine-Tuning is Competitive: A single, well-tuned CAMeL-BERT model was a very close second with a weighted F1-score of 0.597, significantly outperforming its AraBERTv2 counterpart (0.543).
• Model Choice is Crucial: The superior performance of CAMeL-BERT over AraBERTv2 highlights the critical impact of the base model's pre-training on user-generated, dialectal Arabic content.
• Ensembling Can Be Detrimental: The k-fold ensemble method proved detrimental for both base models, suggesting that on small, specialized datasets, averaging predictions can degrade performance.

---

⚙️ Methodology & System Architecture

Our methodology is a three-pronged comparative analysis:

1. Optimized Single Model Fine-Tuning: We used the Optuna framework for automated hyperparameter search to find the best configuration for a single model.

2. K-Fold Ensemble Fine-Tuning: We trained an ensemble of five models using 5-fold cross-validation and averaged their predictions. The training and development sets were combined for this process.

3. In-Context Few-Shot Learning with LLMs: For this strategy, we constructed a prompt that included a task description, category definitions, and 3-5 examples from the training set to guide the LLM in classifying the questions. We utilized several large language models available through the NVIDIA NIM inference microservices API.

A high-level overview of our data processing pipeline, from input data to the final parsed output.

---

📊 Results

Our experiments revealed a clear performance hierarchy among the different approaches. The Palmyra-Med LLM and the single optimized CAMeL-BERT model are the top performers, significantly outclassing all other approaches.

Blind Test Set Performance

Table derived from results presented in the paper.

---

🤖 Pre-trained Models

The pre-trained model weights for our best-performing fine-tuned models can be found on the Hugging Face Hub:

---

---

📜 License & Citation

This project is released under the MIT License. For more details, please refer to the LICENSE file.

If you use our work in your research, please cite our paper:

@inproceedings{mohamed-emad-elden-ab-abukar-2025-sakinah,
    title = "Sakinah-{AI} at {M}ental{QA}: A Comparative Study of Few-Shot, Optimized, and Ensemble Methods for {A}rabic Mental Health Question Classification",
    author = "Mohamed Emad Elden, Fatimah  and
      Ab. Abukar, Mumina",
    editor = "Darwish, Kareem  and
      Ali, Ahmed  and
      Abu Farha, Ibrahim  and
      Touileb, Samia  and
      Zitouni, Imed  and
      Abdelali, Ahmed  and
      Al-Ghamdi, Sharefah  and
      Alkhereyf, Sakhar  and
      Zaghouani, Wajdi  and
      Khalifa, Salam  and
      AlKhamissi, Badr  and
      Almatham, Rawan  and
      Hamed, Injy  and
      Alyafeai, Zaid  and
      Alowisheq, Areeb  and
      Inoue, Go  and
      Mrini, Khalil  and
      Alshammari, Waad",
    booktitle = "Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.arabicnlp-sharedtasks.21/",
    doi = "10.18653/v1/2025.arabicnlp-sharedtasks.21",
    pages = "140--148",
    ISBN = "979-8-89176-356-2",
    abstract = ""
}