Mawqif Track 1 Stance Detection Experiments

Dataset descriptions, local schemas, split statistics, and citation links are available in DATASETS.md.

This repository records experiments for Mawqif V2 Track 1: classify an Arabic post's stance towards a supplied target as Against, Favor, or None.

The development-selection metric is Favg2, the mean of the Favor and Against F1 scores. Favg3 also includes None, but is not the primary checkpoint-selection metric.

Data

The current training scripts use the official Track 1 split:

The original local CSVs, Mawqif_AllTargets_Train.csv and ground_truth.csv, were inspected in data_analytics.ipynb. The fields against_reason, favor_reason, and none_reason are intentionally sparse. CSVs are always loaded with keep_default_na=False, which is important because the label string None must not be interpreted as missing data.

Label balance

The data is materially imbalanced, particularly for None and for particular target/label combinations.

Train-set counts by target:

This imbalance is a likely reason that None remains the hardest class and why target-aware class balancing is worth testing.

Baseline Reference

The supplied benchmark notebook, stanceeval.github.io/StanceEval2026_Baseline_Code.ipynb, fine-tunes aubmindlab/bert-base-arabertv02-twitter as a standard three-class sequence classifier. It encodes (target, text) as a paired input, trains with unweighted cross-entropy and AdamW (lr=2e-5, batch size 32, max length 128, 20 epochs), and retains the checkpoint with the best development Favg2.

Reference development results:

AraBERT-twitter is a BERT-style masked-language-model checkpoint, not a causal language model and not a model with a pretrained stance classification head. Fine-tuning it with AutoModelForSequenceClassification attaches and trains a new task head, which is a valid standard approach.

Experiments

1. Matryoshka encoder with manual pooling

train_track1_matryoshka.py fine-tunes Omartificial-Intelligence-Space/Arabic-Triplet-Matryoshka-V2. It uses the model backbone through AutoModel, mean-pools the final token embeddings with the attention mask, then applies dropout and a three-class linear classifier. Target and post are tokenized as a pair.

Measured result from the best Favg2 checkpoint:

This was below the 83.90 AraBERT-twitter reference baseline. Its None score was also low, so it was not the preferred direction.

2. Matryoshka with AutoModelForSequenceClassification

train_track1_matryoshka_seqclf.py tests the same Matryoshka checkpoint using Hugging Face's standard AutoModelForSequenceClassification interface. This is architecturally valid because the checkpoint has a BERT-compatible backbone; it simply initializes a new sequence-classification head, the same general fine-tuning pattern used with AraBERT.

This script was created to test whether the pooling/head implementation, rather than the encoder itself, limited performance. No result has yet been recorded.

3. Target-specific AraBERT heads

train_track1_arabert_target_heads.py shares one aubmindlab/bert-base-arabertv02-twitter encoder but creates a separate three-way linear classification head for each target:

• Covid Vaccine
• Digital Transformation
• Women empowerment

Each example is still encoded as (target, text), but only the corresponding target head receives its pooled representation. This allows each target's decision boundaries to differ without duplicating the full encoder.

This is the best measured experiment so far.

Overall class-level development scores:

The 85.10 Favg2 improves on the AraBERT-twitter reference baseline by 1.20 points. The model still underperforms on None, and Digital Transformation is its weakest target.

An attempted larger head of Linear -> GELU -> Dropout -> Linear produced lower Favg2 than the simple linear per-target heads. For this data size, the simpler head is the current default.

4. Direction-vector feature and class balancing

train_track1_arabert_target_heads_direction.py extends the target-head model with a scalar stance-direction feature for each example.

For every target, it:

1. Encodes the training examples with a frozen AraBERT encoder.
2. Finds the 10 examples most similar to the class centroid for Favor and the 10 most similar examples for Against.
3. Averages each selected set and computes a normalized direction vector: mean(Favor) - mean(Against).
4. Computes the normalized dot product of a trainable sentence representation with the direction vector for that example's target.
5. Concatenates that scalar to the pooled representation before the target-specific linear head.

This script also introduces editable optimization controls:

• Per-target loss weighting: none, sqrt_inverse, or inverse.
• Optional WeightedRandomSampler, balanced over (target, stance) combinations.
• AdamW weight decay, linear learning-rate scheduling, warmup, label smoothing, and gradient clipping.
• Shorter default training of 8 epochs to make tuning practical.

The default configuration uses sqrt_inverse class weights and does not use the sampler. This experiment is implemented but has not yet produced a recorded validation result.

How To Run

Each training file is a notebook-style Python script with # %% cells and an editable CONFIG dictionary near the top. Run an experiment from the project root, for example:

python train_track1_arabert_target_heads.py

For the direction-feature experiment:

python train_track1_arabert_target_heads_direction.py

Codex zero-shot submission

run_codex_stance_batches.py asks the installed Codex CLI to classify five evaluation examples per request. It supplies only id, tweet_text, and target (not any gold stance field), then appends one Favor, Against, or None label per row to the evaluator-compatible submission file. None is used for neutral, off-topic, or otherwise indeterminate stance. Re-running it resumes from the number of non-empty labels already in that file.

python run_codex_stance_batches.py

The default input is mawqif_evalaution_set.csv and the default output is submissions/codex_calibrated_three_class_results.txt. The calibrated prompt uses None only when there is genuinely no stance after interpreting Arabic dialect, irony, hashtags, and indirect wording. This distinct filename prevents earlier predictions from being reused. To smoke-test one five-row request first:

python run_codex_stance_batches.py --limit 5

One tweet per Codex request

run_codex_stance_one_at_a_time.py uses a concise three-class Arabic stance prompt and sends exactly one tweet in each Codex request. It is resumable and writes to a separate output file by default:

python3 run_codex_stance_one_at_a_time.py --limit 5

Use --lens-file prompts/contextual.txt to add a particular analyst/persona instruction to every one-tweet request. This is useful for testing prompt styles without mixing multiple tweets in the same context.

English tweet translations

run_codex_tweet_translations.py translates one Arabic tweet per Codex request and appends an ID-aligned CSV at translations/codex_tweet_translations.csv. It keeps the original tweet and target alongside translation_en, and resumes safely after an interruption.

python3 run_codex_tweet_translations.py --limit 5

Genuine Arabic-specific idioms and culturally bound expressions are annotated inside the English translation as literal_translation(idiom) <idiom>Arabic phrase | literal English rendering | natural English equivalent</idiom>.

run_codex_translated_stance.py then classifies those English translations, one per Codex request, while preserving the translation CSV order:

python3 run_codex_translated_stance.py --limit 5

It writes submissions/codex_translated_stance_results.txt by default.

Full Arabic contextual prompt, one tweet at a time

run_codex_stance_arabic_contextual.py sends the original Arabic tweets in batches of five using an Arabic translation of the full contextual prompt and Arabic contextual lens. This experiment permits only Favor and Against. It writes a distinct, resumable prediction file:

python3 run_codex_stance_arabic_contextual.py --limit 5

Arabic stance cues plus labels

run_codex_stance_arabic_annotated.py processes five original Arabic tweets per Codex request. In addition to the binary submission labels, it writes a row-aligned JSONL annotation file with target relevance, explicitness, hashtag direction, sarcasm, negation, sentiment toward the target, whether the stance is the author's own, and a short evidence span.

python3 run_codex_stance_arabic_annotated.py --limit 5

The submission labels are written to submissions/codex_arabic_annotated_binary_batch5_results.txt; annotations are written to annotations/codex_arabic_stance_cues.jsonl.

XGBoost using auxiliary annotation labels

train_xgboost_external_labels.py trains a three-class XGBoost classifier on the official Train split and evaluates on Dev using only target, sarcasm, sentiment, and their confidences. It intentionally excludes stance-derived reason fields, since those would leak the gold label.

python3 train_xgboost_external_labels.py

Results, predictions, confusion matrix, and feature importances are written to outputs_xgboost_external_labels/. The opt-in --include-stance-derived-labels mode is an oracle leakage diagnostic, not a valid predictive experiment.

Generating extended cue labels for Train and Dev

annotate_stance_cues.py generates the seven additional Arabic cue labels in five-row Codex batches. It sends only ID, target, and text—not gold stance or reason fields—and safely resumes from a JSONL output file.

python3 annotate_stance_cues.py \
  --input 'stanceeval.github.io/MawqifV2/Track 1/train.csv' \
  --output annotations/train_stance_cues.jsonl

python3 annotate_stance_cues.py \
  --input 'stanceeval.github.io/MawqifV2/Track 1/dev.csv' \
  --output annotations/dev_stance_cues.jsonl

The same script also supports CSVs using an unnamed ID column and topic instead of target, such as the added multi-topic Arabic stance data. Its provided stance field is deliberately not sent to Codex:

python3 annotate_stance_cues.py \
  --input training_women_driving_arabic_tweets.csv \
  --output annotations/topic_stance_train_cues.jsonl

python3 annotate_stance_cues.py \
  --input test_women_driving_arabic_tweets.csv \
  --output annotations/topic_stance_test_cues.jsonl

prepare_arabicstancex_women_driving.py converts those cue JSONL files to CSV (when present) and writes clean ArabicStanceX Women Driving subsets with id, text, stance, and topic columns:

python3 prepare_arabicstancex_women_driving.py

Outputs are written under processed_data/arabicstancex/.

Enhanced Women Driving cue annotation

annotate_women_driving_enhanced_cues.py sends the Women Driving subset to Codex in batches of five. It preserves each source tweet in the JSONL output, does not send the gold stance label, and adds the usual cue set plus hashtag_usage_relation, irony_type, and negation_scope.

python3 annotate_women_driving_enhanced_cues.py \
  --input processed_data/arabicstancex/women_driving_train.csv \
  --output annotations/women_driving_train_enhanced_cues.jsonl

python3 annotate_women_driving_enhanced_cues.py \
  --input processed_data/arabicstancex/women_driving_test.csv \
  --output annotations/women_driving_test_enhanced_cues.jsonl

Use --limit 5 for a single five-tweet trial; rerunning resumes safely.

To extend the existing Mawqif evaluation annotations without relabeling their original seven cues, run:

python3 augment_evaluation_stance_cues.py

It creates annotations/evaluation_stance_enhanced_cues.jsonl and adds the three enhanced cues plus the original tweet text.

The same augmenter supports a target-filtered Mawqif split. For Women Empowerment (1,190 Train rows and 210 Dev rows):

python3 augment_evaluation_stance_cues.py \
  --input Mawqif_AllTargets_Train.csv \
  --existing-cues annotations/train_stance_cues.jsonl \
  --target 'Women empowerment' \
  --output annotations/women_empowerment_train_enhanced_cues.jsonl

python3 augment_evaluation_stance_cues.py \
  --input ground_truth.csv \
  --existing-cues annotations/dev_stance_cues.jsonl \
  --target 'Women empowerment' \
  --output annotations/women_empowerment_dev_enhanced_cues.jsonl

For parallel runs, run_women_empowerment_parallel.py launches 12 independent train workers (eleven 100-row shards and one 90-row shard) or two 105-row Dev workers. Each worker calls Codex in batches of five and writes its own JSONL; after all workers succeed, the strict merger validates every ID occurs once and writes the ordered final file.

python3 run_women_empowerment_parallel.py --split train
python3 run_women_empowerment_parallel.py --split dev

Add --dry-run to inspect shard ranges without launching Codex.

Weak evaluation auxiliary labels

run_evaluation_auxiliary_parallel.py creates Codex-derived (not gold) Mawqif-style sentiment, sarcasm, sentiment:confidence, and sarcasm:confidence fields for all evaluation tweets. It uses five-tweet batches, writes independently resumable shards, and merges only after full ID coverage is verified. The default is four parallel workers; choose another count with --workers.

python3 run_evaluation_auxiliary_parallel.py

The fused output is annotations/evaluation_weak_auxiliary.jsonl and carries annotation_source: "codex_weak" so it cannot be mistaken for gold labels.

Cross-dataset Women Driving normalization

normalize_women_driving_domains.py creates deterministic shared-space text variants for Mawqif evaluation and ArabicStanceX Women Driving Train/Test. It does not paraphrase tweets with a generative model, so it cannot rewrite a stance, quoted slogan, or irony. It consistently normalizes URLs, mentions, emoji sequences, elongation, Arabic spelling variants, and campaign hashtags. With the enhanced cue JSONLs, hashtags become generic direction tokens such as [HASHTAG_PRO], [HASHTAG_ANTI], or [HASHTAG_TOPIC]; fully masked and dropped-hashtag variants are also supplied for source-training augmentation.

The additional normalized_text_extended* variants apply five conservative domain-style rules: canonicalize common Women Driving phrases to [WOMEN_DRIVING], turn laughter/elongation into [LAUGHTER] or a short form, retain at most one [EMOJI], remove retweet/reply headers and repeated punctuation, and mask a deliberately small public-figure list as [PUBLIC_FIGURE]. They are separate columns for ablation rather than a replacement for the less-modified text.

python3 normalize_women_driving_domains.py

Outputs are written under processed_data/shared_women_driving/.

To quantify whether normalization reduces dataset-specific wording, run the reproducible five-fold domain classifier comparison:

python3 evaluate_women_driving_domain_shift.py

It compares raw text with cue-token, masked-hashtag, and dropped-hashtag representations. Metrics and out-of-fold predictions are written to outputs_domain_shift_normalized/.

Parallel contextual stance prediction with weak auxiliary labels

run_evaluation_weak_auxiliary_stance_parallel.py classifies the Mawqif evaluation set with the binary Arabic contextual prompt in prompts/arabic_contextual_weak_auxiliary_binary.txt. Each five-tweet Codex batch also receives the aligned Codex weak sentiment, sarcasm, and two confidence values. Twelve independent workers write JSONL shards, which are strictly checked for unique, complete IDs before being fused in original CSV order into an evaluator-compatible submission file.

python3 run_evaluation_weak_auxiliary_stance_parallel.py

Use --dry-run to inspect shards first or --workers N to change concurrency.

Aggressive binary correction of contextual predictions

run_evaluation_stance_aggressive_correction_parallel.py re-evaluates each codex_contextual_results.txt prediction independently before comparing it to the existing label. Unlike the conservative corrector, it flips whenever the opposite binary direction is better supported by the tweet; it has no None diagnostic and no preset correction quota. It records evidence and flip confidence, then reports Favor→Against and Against→Favor counts in outputs_correction/codex_contextual_aggressive_correction_summary.json.

python3 run_evaluation_stance_aggressive_correction_parallel.py

Parallel conservative correction of a binary submission

run_evaluation_stance_correction_parallel.py reviews the aligned codex_contextual_results.txt baseline with an Arabic conservative-correction prompt. It uses five-tweet batches across twelve independent workers. The final merge checks every ID and baseline label, writes the corrected evaluator-format submission, a row-level audit JSONL, and a JSON summary with Favor→Against and Against→Favor counts.

python3 run_evaluation_stance_correction_parallel.py

The correction prompt is in prompts/arabic_conservative_stance_corrector_binary.txt.

Zero-shot Arabic contextual baseline, ten tweets per request

run_codex_stance_arabic_contextual_zeroshot_batch10.py applies the existing Arabic contextual system prompt to ten evaluation tweets per Codex request. It uses no few-shot examples, weak labels, or cue annotations and writes a binary Favor/Against submission in source order.

python3 run_codex_stance_arabic_contextual_zeroshot_batch10.py

Ten-member Arabic contextual self-consistency ensemble

run_codex_arabic_contextual_ensemble10.py sends each ten-tweet batch to ten independent zero-shot Arabic contextual Codex replicas. Each replica returns a three-class label and self-confidence. The final label is the majority vote; the reported ensemble confidence is the winning vote share, while mean_member_confidence records the average self-confidence of members that supported that label.

python3 run_codex_arabic_contextual_ensemble10.py

The evaluator-format labels are written to submissions/codex_arabic_contextual_ensemble10_majority_results.txt; detailed votes and confidence values are written to the adjacent JSONL file.

Targeted three-lens disagreement adjudication

run_codex_targeted_adjudication.py preserves the contextual binary prediction where the contextual, ensemble, and weak-auxiliary systems agree. Only their disagreement rows are sent to three independent Arabic adjudication lenses: quote/negation/hashtag adoption, sarcasm/pragmatics, and direct argument/entailment. Each lens makes an independent binary decision before examining the competing predictions; the three final lens decisions are majority-fused for those candidates only.

python3 run_codex_targeted_adjudication.py

It writes the final labels to submissions/codex_targeted_adjudication_results.txt and a candidate-level decision audit under annotations/.

Gemini Arabic-contextual baseline

run_evaluation_gemini_arabic_contextual_parallel.py reproduces the binary Arabic contextual stance workflow using the official google-genai SDK, Gemini structured JSON output, five-tweet batches, resumable shards, and strict ordered fusion. It defaults to gemini-2.5-pro for accuracy-oriented Arabic dialect/sarcasm interpretation; this is not an Arabic-specialized model.

pip install --upgrade google-genai
export GEMINI_API_KEY='...'
python3 run_evaluation_gemini_arabic_contextual_parallel.py

For Vertex AI instead of the Developer API, authenticate with Application Default Credentials and run with --backend vertex --project PROJECT_ID (optionally --location global). The final submission is submissions/gemini_2_5_pro_arabic_contextual_results.txt.

XGBoost: unseen-target evaluation with shared cue labels

train_xgboost_unseen_target.py combines labeled Train and Dev rows, then performs leave-one-target-out evaluation. For each of the three official targets, it trains only on the other two targets and tests on the held-out target. It uses the same seven Codex cue fields for every split and excludes target identity and evidence text.

python3 train_xgboost_unseen_target.py

The per-target predictions, confusion matrices, importances, and summary are written to outputs_xgboost_unseen_target/.

To train on all Train targets and test on all Dev topics instead:

python3 train_xgboost_unseen_target.py --mode train-dev \
  --output-dir outputs_xgboost_cues_train_dev

Local Arabic-to-English translations (OPUS-MT)

translate_tweets_opus_mt.py translates all mawqif_evaluation*.csv files with Helsinki-NLP/opus-mt-tc-big-ar-en, writes resumable CSVs under translations/opus_mt_ar_en/, and uses CUDA automatically when available. The supplied evaluation filename has the legacy spelling mawqif_evalaution_set.csv; the script recognizes it when run with its default arguments.

pip install torch transformers sentencepiece
python3 translate_tweets_opus_mt.py

Five-lens Codex ensemble

run_codex_stance_ensemble.py sends each five-row batch to five independent, objective stance-analysis prompt profiles: literal, contextual, irony-aware, argument-focused, and conservative/evidence-bound. It does not show any gold labels to Codex. It records each profile's labels separately and appends a five-vote ensemble label to the ensemble file. None requires a strict 3/5 vote majority; otherwise the ensemble selects between Favor and Against.

python3 run_codex_stance_ensemble.py --limit 5

The files created under submissions/ are:

• codex_explicit_calibrated_three_class_results.txt
• codex_contextual_calibrated_three_class_results.txt
• codex_sarcasm_calibrated_three_class_results.txt
• codex_argument_calibrated_three_class_results.txt
• codex_conservative_calibrated_three_class_results.txt
• codex_ensemble_calibrated_three_class_results.txt (the majority-vote submission)

All six files must have the same number of labels to resume. This prevents a partially written persona file from silently misaligning future votes.

Iterating from stored binary submissions

build_contextual_iterations.py creates reproducible candidate submissions from the saved two-class persona predictions in stored_submissions/. The first iteration retains the best-scoring contextual submission as its anchor and changes a row only when all four other personas unanimously select the opposite label. This avoids encoding hidden class totals or forcing label quotas.

python3 build_contextual_iterations.py --iteration 1

It writes a submission and a JSON manifest to submissions/iterations/. After evaluating that single candidate, record its leaderboard Favg2 locally:

python3 build_contextual_iterations.py --iteration 1 --record-favg2 0.000000

Only if the first iteration improves should the looser iteration 2 (three of the four non-contextual personas agree) be evaluated.

Each run selects the best checkpoint by development Favg2 and writes its training history, checkpoint, per-target results, and development predictions inside the configured output_dir.

Input And Preprocessing Ablation

run_track1_arabert_ablations.py performs a controlled 27-run comparison while keeping the target-head model, optimizer, seed, and checkpoint criterion unchanged:

• Preprocessing: raw, benchmark, and arabert_official.
• Maximum length: 64, 96, and 128 tokens.
• Input: tweet_only, (English target, tweet), or (Arabic target, tweet).

The benchmark mode is the character-level cleaning used in the supplied baseline notebook: remove Arabic diacritics and non-Arabic characters, cap character repetitions at two, and collapse whitespace. The existing target-head script had used this exact mode, so it should not be treated as a separate unmeasured "custom" preprocessing baseline.

The arabert_official mode uses arabert.preprocess.ArabertPreprocessor and requires the optional dependency:

pip install arabert
python run_track1_arabert_ablations.py

The Arabic target mappings are Covid Vaccine -> لقاح كوفيد, Digital Transformation -> التحول الرقمي, and Women empowerment -> تمكين المرأة. These are intentionally passed as the first sequence in the tokenizer pair, rather than as English text concatenated to an Arabic tweet. The complete ranking is written incrementally to outputs_track1_arabert_ablations/ablation_results.csv, so completed results survive a later failed run.

Recommended Optimization Sequence

Keep the current 85.10 target-head script as the control and change one factor at a time. A practical sequence is:

1. Run the direction-feature script with class_weight_mode="none" and no sampler to isolate the direction feature.
2. Compare sqrt_inverse and inverse target-specific class weights, still with no sampler.
3. Test use_weighted_sampler=True separately; do not combine it with aggressive inverse weights initially, because both can over-correct the minority class.
4. Sweep learning rate over 1e-5, 1.5e-5, 2e-5, and 3e-5 with epochs in 4, 6, and 8.
5. Sweep weight decay over 0.0, 0.01, and 0.05, and warmup ratio over 0.06 and 0.10.
6. Use multiple seeds for promising settings and compare mean and variance of Favg2, rather than trusting one run.

The central optimization target is improving Against without sacrificing Favor; None should be monitored through Favg3, but it is not part of the official Favg2 selection metric.

Multi-Encoder LoRA Fusion

train_track1_multi_encoder_lora_fusion.py trains independent LoRA adapters for AraBERT-twitter, MARBERT, and Arabic-Triplet-Matryoshka-V2. It projects their pooled representations into a shared 256-dimensional space, combines them with learned target-specific source weights, then applies the existing target-specific classifiers. It does not merge LoRA weights across incompatible base models.

pip install peft
python train_track1_multi_encoder_lora_fusion.py

The default micro-batch is 4 with four gradient-accumulation steps. Results include fusion_weights_by_target.csv, which shows how much each target uses each source encoder.

AraBERT-Twitter Women Driving Domain Adaptation

train_arabert_twitter_women_driving_dann.py fine-tunes aubmindlab/bert-base-arabertv02-twitter as a target-aware three-class model on ArabicStanceX Women Driving data. It uses a raw-versus-masked tweet consistency loss and writes both source-test metrics and a Mawqif submission.

# Source-supervised baseline
python3 train_arabert_twitter_women_driving_dann.py

# Transductive DANN variant: only if use of unlabelled evaluation tweets is permitted.
python3 train_arabert_twitter_women_driving_dann.py --use-target-unlabeled

The transductive mode adds a gradient-reversal domain discriminator over ArabicStanceX and unlabelled Mawqif tweets. It must not be used when benchmark rules prohibit adaptation on evaluation inputs.