This repository contains the code and detailed analysis regarding system paper in HalluScoring Shared Task Arabic NLP conference colocated with EMNLP.
Jupyter Notebook
0
2 commits
updated Aug 21, 2026
Team AyahVerse's submission to the HalluScoring 2026 shared task (ArabicNLP 2026): detecting factual hallucinations in Arabic LLM outputs and recovering the correct answer.
| Track | Task | What It Does |
|---|---|---|
| Track 1 | Hallucination Detection | Binary classification: is an LLM-generated Arabic answer hallucinated (0/1)? |
| Track 2 | Detection + Truth Recovery | Detect hallucination AND select the correct answer from 6 options (A–F) |
github_release/
├── README.md # This file
├── Task-1/ # Track 1 (hallucination detection)
└── Task-2/ # Track 2 (detection + option selection)
| Task | Result |
|---|---|
| Track 1.1 (Dev / Test) | AUC-ROC 0.922 / 0.763 |
| Track 1.2 (Dev / Test) | AUC-ROC 0.749 / 0.729 |
| Track 2.1 (Combined) | 0.784 |
| Track 2.2 (Combined) | 0.829 |
Goal: Given a question + an LLM-generated answer, predict whether the answer is hallucinated.
Key idea: Fine-tune CamelBERT with:
| File | Description |
|---|---|
train_track1.py | Train CamelBERT (multi-layer supervision + category embedding) |
predict_track1.py | Run inference + evaluate (AUC-ROC, F1-Macro, AUC-PR) |
ablation_experiments.ipynb | Colab notebook: component ablation experiments |
submission_tracker1_1.ipynb | Submission notebook for Task 1.1 |
model_weights/ | Trained CamelBERT weights (safetensors) |
README_Submission_Task1.md | Submission guide |
# Train (full model: layer supervision + category embedding)
python train_track1.py --task 1.1 --models camelbert --supervise_layers 4,8,12 --category_embedding
# Predict / evaluate
python predict_track1.py --task 1.1 --test_path task1.1_data/task1.1_dev.xlsx --has_labels
Goal: Two-step pipeline:
Key idea: Decouple the two sub-problems into specialized models:
Both models run on every example; outputs merged into {pred_label, pred_option}.
Metric: combined = 0.4 × Macro-F1 + 0.6 × Option Accuracy
| File | Description |
|---|---|
train_detector_bert.py | Train CamelBERT detection model (Task 2) |
train_option_selector_acegpt.py | Train AceGPT option selector (QLoRA) |
baseline_task2.1.ipynb | Simple baseline evaluation (no training) — Task 2.1 |
baseline_task2.2.ipynb | Simple baseline evaluation (no training) — Task 2.2 |
submission_and_evaluation.ipynb | Load trained models, run inference, full leaderboard metrics |
# Train CamelBERT detector (Task 2.1)
python train_detector_bert.py \
--train_path data/islamic/train_subtask3-1.json \
--dev_path data/islamic/dev_subtask3-1.json \
--output_dir ./outputs_bert_detector
# Train AceGPT option selector (Task 2.1)
python train_option_selector_acegpt.py \
--train_path data/islamic/train_subtask3-1.json \
--dev_path data/islamic/dev_subtask3-1.json \
--output_dir ./outputs_acegpt_options
For Task 2.2, use the data/general_culture/ files (dev_subtask2-2.json = labeled dev, dev_subtask3-2.json = unlabeled test).
torch
transformers
datasets
accelerate
scikit-learn
scipy
pandas
openpyxl
safetensors
bitsandbytes
peft
Team AyahVerse at HalluScoring 2026. If you use this code, please cite our paper (released with the shared task proceedings).
[email protected]This repository contains the code and detailed analysis regarding system paper in HalluScoring Shared Task Arabic NLP conference colocated with EMNLP.
Jupyter Notebook
0
2 commits
updated Aug 21, 2026
Team AyahVerse's submission to the HalluScoring 2026 shared task (ArabicNLP 2026): detecting factual hallucinations in Arabic LLM outputs and recovering the correct answer.
| Track | Task | What It Does |
|---|---|---|
| Track 1 | Hallucination Detection | Binary classification: is an LLM-generated Arabic answer hallucinated (0/1)? |
| Track 2 | Detection + Truth Recovery | Detect hallucination AND select the correct answer from 6 options (A–F) |
github_release/
├── README.md # This file
├── Task-1/ # Track 1 (hallucination detection)
└── Task-2/ # Track 2 (detection + option selection)
| Task | Result |
|---|---|
| Track 1.1 (Dev / Test) | AUC-ROC 0.922 / 0.763 |
| Track 1.2 (Dev / Test) | AUC-ROC 0.749 / 0.729 |
| Track 2.1 (Combined) | 0.784 |
| Track 2.2 (Combined) | 0.829 |
Goal: Given a question + an LLM-generated answer, predict whether the answer is hallucinated.
Key idea: Fine-tune CamelBERT with:
| File | Description |
|---|---|
train_track1.py | Train CamelBERT (multi-layer supervision + category embedding) |
predict_track1.py | Run inference + evaluate (AUC-ROC, F1-Macro, AUC-PR) |
ablation_experiments.ipynb | Colab notebook: component ablation experiments |
submission_tracker1_1.ipynb | Submission notebook for Task 1.1 |
model_weights/ | Trained CamelBERT weights (safetensors) |
README_Submission_Task1.md | Submission guide |
# Train (full model: layer supervision + category embedding)
python train_track1.py --task 1.1 --models camelbert --supervise_layers 4,8,12 --category_embedding
# Predict / evaluate
python predict_track1.py --task 1.1 --test_path task1.1_data/task1.1_dev.xlsx --has_labels
Goal: Two-step pipeline:
Key idea: Decouple the two sub-problems into specialized models:
Both models run on every example; outputs merged into {pred_label, pred_option}.
Metric: combined = 0.4 × Macro-F1 + 0.6 × Option Accuracy
| File | Description |
|---|---|
train_detector_bert.py | Train CamelBERT detection model (Task 2) |
train_option_selector_acegpt.py | Train AceGPT option selector (QLoRA) |
baseline_task2.1.ipynb | Simple baseline evaluation (no training) — Task 2.1 |
baseline_task2.2.ipynb | Simple baseline evaluation (no training) — Task 2.2 |
submission_and_evaluation.ipynb | Load trained models, run inference, full leaderboard metrics |
# Train CamelBERT detector (Task 2.1)
python train_detector_bert.py \
--train_path data/islamic/train_subtask3-1.json \
--dev_path data/islamic/dev_subtask3-1.json \
--output_dir ./outputs_bert_detector
# Train AceGPT option selector (Task 2.1)
python train_option_selector_acegpt.py \
--train_path data/islamic/train_subtask3-1.json \
--dev_path data/islamic/dev_subtask3-1.json \
--output_dir ./outputs_acegpt_options
For Task 2.2, use the data/general_culture/ files (dev_subtask2-2.json = labeled dev, dev_subtask3-2.json = unlabeled test).
torch
transformers
datasets
accelerate
scikit-learn
scipy
pandas
openpyxl
safetensors
bitsandbytes
peft
Team AyahVerse at HalluScoring 2026. If you use this code, please cite our paper (released with the shared task proceedings).
[email protected]