Arabic Stance Detection for Unseen Targets (StanceEval-2026)

This repository contains the source code and experimental setup for our submission to StanceEval-2026: Track 2, focusing on zero-shot and cross-target Arabic stance detection.

The architecture utilizes a prompt-based Masked Language Modeling (MLM) approach with AraBERT, designed to generalize to completely unseen topics using the Mawqif-v2 dataset.

🎯 Task Overview

• Track: StanceEval-2026 Track 2 (Unseen Targets)
• Objective: Classify the stance of an Arabic tweet toward a target topic that was not seen during training.
• Labels: Favor, Against, or None.
• Primary Metric: $F_{avg2}$ (Macro-average $F_1$ over Favor and Against, excluding None).
• Competitive Baseline: Qwen 2.5 (72B, zero-shot) at 74.75 $F_{avg2}$.

🧠 Architecture & Methodology

Our approach frames stance detection as a text-to-text fill-in-the-blank task to maximize the zero-shot capabilities of pre-trained language models.

• Prompt-Based MLM: We use aubmindlab/bert-large-arabertv02-twitter to predict the masked stance label based on the text, target, and auxiliary sentiment labels.
• Synthetic Data Augmentation: To handle class imbalance, we dynamically generate synthetic None examples by permuting the target topics across the training set (target mismatching).
• Ensemble Strategy: We train multiple model variations (Base, Augmented None, Class-Weighted) across different random seeds. The final predictions are aggregated using:
  • Soft Voting: Averaging predicted probabilities across all models.
  • Hard Voting: Majority rule voting per instance.

🗂️ Dataset & Preprocessing

The model is trained and evaluated on the Mawqif-v2 dataset.

• Seen Targets (Train): COVID-19 Vaccine, Digital Transformation (2,721 tweets).
• Unseen Targets (Dev/Test): Held-out for cross-target evaluation.

Text Preprocessing Pipeline:

1. Strips Arabic diacritics (Tashkeel).
2. Removes non-Arabic characters while preserving numbers.
3. Caps repeated characters to a maximum of two (handling social media elongation).
4. Normalizes multi-spaces.

🚀 Repository Structure

├── datasets/
│   ├── stance/MawqifV2/       # Contains train.csv and dev.csv
│   └── stance/STANCE_X/       # Intermediate/Pre-training datasets
├── preprocess_data.py         # Text cleaning and synthetic 'None' augmentation
├── evaluator.py               # Favg2, Favg3, and accuracy metric computations
└── train_for_submission.py    # Training loop, data loaders, and ensemble voting

⚙️ Setup & Usage

1. Requirements

Ensure you have Python 3.8+ installed. Install the required dependencies using the provided requirements.txt file:

pip install -r requirements.txt

2. Run the Training Pipeline

The main script handles pre-training, fine-tuning, and evaluation across multiple seeds and configurations.

python train_for_submission.py

Note: Ensure your datasets are placed in the correct directories as defined in the BASE_DIR variable before running the script.

---

Disclaimer: This README was generated by AI.