Aspect-Based Sentiment Analysis on Arabic customer reviews.
For each review the model picks the aspects that are mentioned (from the 9 allowed classes) and attaches a sentiment to each one.
Input: "الأكل حلو بس الخدمة بطيئة والسعر غالي"
Output: food: positive, service: negative, price: negative
Allowed aspects: food, service, price, cleanliness, delivery, ambiance, app_experience, general, none.
Allowed sentiments: positive, negative, neutral.
| Metric | Validation | Hidden test |
|---|---|---|
| Micro-F1 | 0.8246 | 0.832 |
| Precision | 0.835 | 0.832 |
| Recall | 0.814 | 0.833 |
| Pillar 1 (out of 30) | — | 24.97 |
The shipped submission.json was produced with the exact pipeline in
make_final_submission.py using the three trained checkpoints.
Everything you need is inside this folder. Install requirements, point it at the hidden-test xlsx, one command:
pip install -r requirements.txt
python make_final_submission.py --input path/to/hidden_test_nolabels.xlsx --out submission.json
~5 minutes on CPU. Writes submission.json next to the script.
The command reproduces submission.json that ships with this zip — the
same JSON we submitted for Pillar 1. No flags needed.
make_final_submission.py runs a four-step recipe:
1. Score with each checkpoint. We ship three trained runs under
checkpoints/run1, run2, run3. Every run has the same MARBERTv2 +
9-head architecture but differs in training: run1 is the baseline on
the labeled data only, run2 adds 2,954 confident pseudo-labels from
the unlabeled pool with multi-sample dropout, run3 is a third seed
with a different random initialisation. Each produces a [N, 9, 4]
tensor of probabilities — one 4-way distribution
(absent / positive / negative / neutral) per aspect per review.
2. Per-aspect weighted ensemble. On validation we measured which of
the three runs is strongest for each aspect independently and saved the
mapping to checkpoints/per_aspect_models.json. At inference the
best run for that aspect gets 85% of the vote and the other two split
the remaining 15% equally. This beats uniform averaging because
different seeds/training configs specialise at different labels —
run1 is best on food, app_experience, none; run3 wins most of
the rest.
3. Confidence filter. Any predicted state whose max softmax
probability is below 0.60 is pushed toward absent (we add a strong
bias to its absent logit). That keeps only the high-conviction calls
and was the single biggest post-hoc win over a plain ensemble —
roughly +0.5 F1 on our experiments because it strips noisy
predictions without touching the confident ones.
4. Latin-script keyword nudge. A small lexicon
(excellent / parfait / great / mauvais / ..., plus aspect cue words
like food, service, delivery) shifts the sentiment logits when
the review is mostly written in Latin script. MARBERTv2 is
Arabic-first and occasionally hedges on French/English rows; the
nudge corrects the obvious cases without touching Arabic predictions.
The output goes through the same JSON encoder we use elsewhere —
review_id, aspects list, aspect_sentiments dict with matching
keys — and validate_submission.py lints the result against the
schema.
absa/
constants.py # label taxonomy + integer mappings
data.py # Dataset class, collate, class weights
inference.py # decode probabilities → submission JSON
metrics.py # micro-F1 over (review_id, aspect, sentiment)
model.py # ABSAModel = MARBERTv2 + 9 heads
multilingual.py # keyword nudge for Latin-script reviews
paths.py # find xlsx files regardless of CWD
star_bias.py # star-rating prior (tried, not used in final)
text.py # normalisation + meta-prefix input
training.py # train loop with AdamW + AMP + MSD
tta.py # test-time augmentation variants (unused in final)
train.py # train a model (baseline or with pseudo-labels)
pseudolabel.py # score unlabeled pool, keep confident rows
tune_final.py # (re)search per-aspect inference config on val
make_final_submission.py # produce submission.json
app.py # Gradio demo UI
validate_submission.py # lint submission.json against the schema
analyze_stats.py # EDA dump — labels, platforms, lengths
make_plots.py # regenerate the 4 PNGs in docs/
checkpoints/ # shipped weights
run1/best.pt # baseline, labeled only
run2/best.pt # + pseudo-labels, multi-sample dropout
run3/best.pt # + different random seed
per_aspect_models.json # which run to prefer for each aspect
pseudo.parquet # 2954 confident pseudo-labels used by run2/3
docs/ # EDA plots (aspect freq, length, etc.)
submission.json # the JSON we submitted for Pillar 1
run_pipeline.sh / .bat # one-shot reproduction from scratch
requirements.txt
README.md # this file
These are the exact commands we ran, in order, on CPU. Pass --batch 16
on a real GPU.
# 1. Put the xlsx files somewhere findable — either ./data/ or pass --train/--val.
# 2. Baseline — labeled data only, 4 epochs.
python train.py --out checkpoints/run1 --epochs 4 --batch 8 --max_len 128
# 3. Pseudo-label the 7k unlabeled pool at ≥0.85 confidence per aspect.
python pseudolabel.py --ckpt checkpoints/run1/best.pt --out checkpoints/pseudo.parquet --threshold 0.85
# 4. Retrain on labeled + pseudo with multi-sample dropout.
python train.py --pseudo checkpoints/pseudo.parquet --out checkpoints/run2 \
--epochs 5 --batch 8 --max_len 128 --msd 4
# 5. Third seed for ensemble diversity.
python train.py --pseudo checkpoints/pseudo.parquet --out checkpoints/run3 \
--epochs 5 --batch 8 --max_len 128 --msd 4 --seed 123
# 6. Produce the submission.
python make_final_submission.py --input hidden_test_nolabels.xlsx --out submission.json
End-to-end: about 3 hours on a modern 8-core CPU, well under 30 minutes on a small CUDA GPU.
The backbone is MARBERTv2 — BERT pretrained on 1B dialectal-Arabic
tweets. For Egyptian slang it generalises significantly better than
plain mBERT (~+4 F1 on our validation). On top of the encoder we
mean-pool over the non-padding tokens and feed the pooled vector into
9 small classification heads, one per aspect. Each head emits a
4-way softmax over {absent, positive, negative, neutral}. This frames
presence-detection and sentiment together as a single choice, which
saves us from the classic mismatch where aspect detection fires but
sentiment hedges.
The input isn't just the raw review — we prepend
[platform|category|Nstar]. Short reviews ⇒ any cheap prior helps,
and platform + category are strong signals on their own (play_store
implies app_experience, restaurant implies food).
Loss is inverse-frequency weighted cross-entropy per (aspect, state)
bin — the dataset is heavily skewed (service is half the rows, none is
3%, neutral is 4.5% of all sentiments) so without weights the model just
predicts service.
Training config that mattered: AdamW with 2e-5 on the backbone and 1e-4 on the heads, 10% warmup then linear decay, gradient clip at 1.0, multi-sample dropout (4 samples averaged) on the runs trained with pseudo-labels.
pip install gradio
python app.py
Auto-picks the latest checkpoint; paste a review, hit Analyze.
multilingual.py
covered the common cases.none on
single-word greetings, force app_experience=negative on play_store
MIT on the code. Weights inherit MARBERTv2's upstream terms. "# NLP"
Aspect-Based Sentiment Analysis on Arabic customer reviews.
For each review the model picks the aspects that are mentioned (from the 9 allowed classes) and attaches a sentiment to each one.
Input: "الأكل حلو بس الخدمة بطيئة والسعر غالي"
Output: food: positive, service: negative, price: negative
Allowed aspects: food, service, price, cleanliness, delivery, ambiance, app_experience, general, none.
Allowed sentiments: positive, negative, neutral.
| Metric | Validation | Hidden test |
|---|---|---|
| Micro-F1 | 0.8246 | 0.832 |
| Precision | 0.835 | 0.832 |
| Recall | 0.814 | 0.833 |
| Pillar 1 (out of 30) | — | 24.97 |
The shipped submission.json was produced with the exact pipeline in
make_final_submission.py using the three trained checkpoints.
Everything you need is inside this folder. Install requirements, point it at the hidden-test xlsx, one command:
pip install -r requirements.txt
python make_final_submission.py --input path/to/hidden_test_nolabels.xlsx --out submission.json
~5 minutes on CPU. Writes submission.json next to the script.
The command reproduces submission.json that ships with this zip — the
same JSON we submitted for Pillar 1. No flags needed.
make_final_submission.py runs a four-step recipe:
1. Score with each checkpoint. We ship three trained runs under
checkpoints/run1, run2, run3. Every run has the same MARBERTv2 +
9-head architecture but differs in training: run1 is the baseline on
the labeled data only, run2 adds 2,954 confident pseudo-labels from
the unlabeled pool with multi-sample dropout, run3 is a third seed
with a different random initialisation. Each produces a [N, 9, 4]
tensor of probabilities — one 4-way distribution
(absent / positive / negative / neutral) per aspect per review.
2. Per-aspect weighted ensemble. On validation we measured which of
the three runs is strongest for each aspect independently and saved the
mapping to checkpoints/per_aspect_models.json. At inference the
best run for that aspect gets 85% of the vote and the other two split
the remaining 15% equally. This beats uniform averaging because
different seeds/training configs specialise at different labels —
run1 is best on food, app_experience, none; run3 wins most of
the rest.
3. Confidence filter. Any predicted state whose max softmax
probability is below 0.60 is pushed toward absent (we add a strong
bias to its absent logit). That keeps only the high-conviction calls
and was the single biggest post-hoc win over a plain ensemble —
roughly +0.5 F1 on our experiments because it strips noisy
predictions without touching the confident ones.
4. Latin-script keyword nudge. A small lexicon
(excellent / parfait / great / mauvais / ..., plus aspect cue words
like food, service, delivery) shifts the sentiment logits when
the review is mostly written in Latin script. MARBERTv2 is
Arabic-first and occasionally hedges on French/English rows; the
nudge corrects the obvious cases without touching Arabic predictions.
The output goes through the same JSON encoder we use elsewhere —
review_id, aspects list, aspect_sentiments dict with matching
keys — and validate_submission.py lints the result against the
schema.
absa/
constants.py # label taxonomy + integer mappings
data.py # Dataset class, collate, class weights
inference.py # decode probabilities → submission JSON
metrics.py # micro-F1 over (review_id, aspect, sentiment)
model.py # ABSAModel = MARBERTv2 + 9 heads
multilingual.py # keyword nudge for Latin-script reviews
paths.py # find xlsx files regardless of CWD
star_bias.py # star-rating prior (tried, not used in final)
text.py # normalisation + meta-prefix input
training.py # train loop with AdamW + AMP + MSD
tta.py # test-time augmentation variants (unused in final)
train.py # train a model (baseline or with pseudo-labels)
pseudolabel.py # score unlabeled pool, keep confident rows
tune_final.py # (re)search per-aspect inference config on val
make_final_submission.py # produce submission.json
app.py # Gradio demo UI
validate_submission.py # lint submission.json against the schema
analyze_stats.py # EDA dump — labels, platforms, lengths
make_plots.py # regenerate the 4 PNGs in docs/
checkpoints/ # shipped weights
run1/best.pt # baseline, labeled only
run2/best.pt # + pseudo-labels, multi-sample dropout
run3/best.pt # + different random seed
per_aspect_models.json # which run to prefer for each aspect
pseudo.parquet # 2954 confident pseudo-labels used by run2/3
docs/ # EDA plots (aspect freq, length, etc.)
submission.json # the JSON we submitted for Pillar 1
run_pipeline.sh / .bat # one-shot reproduction from scratch
requirements.txt
README.md # this file
These are the exact commands we ran, in order, on CPU. Pass --batch 16
on a real GPU.
# 1. Put the xlsx files somewhere findable — either ./data/ or pass --train/--val.
# 2. Baseline — labeled data only, 4 epochs.
python train.py --out checkpoints/run1 --epochs 4 --batch 8 --max_len 128
# 3. Pseudo-label the 7k unlabeled pool at ≥0.85 confidence per aspect.
python pseudolabel.py --ckpt checkpoints/run1/best.pt --out checkpoints/pseudo.parquet --threshold 0.85
# 4. Retrain on labeled + pseudo with multi-sample dropout.
python train.py --pseudo checkpoints/pseudo.parquet --out checkpoints/run2 \
--epochs 5 --batch 8 --max_len 128 --msd 4
# 5. Third seed for ensemble diversity.
python train.py --pseudo checkpoints/pseudo.parquet --out checkpoints/run3 \
--epochs 5 --batch 8 --max_len 128 --msd 4 --seed 123
# 6. Produce the submission.
python make_final_submission.py --input hidden_test_nolabels.xlsx --out submission.json
End-to-end: about 3 hours on a modern 8-core CPU, well under 30 minutes on a small CUDA GPU.
The backbone is MARBERTv2 — BERT pretrained on 1B dialectal-Arabic
tweets. For Egyptian slang it generalises significantly better than
plain mBERT (~+4 F1 on our validation). On top of the encoder we
mean-pool over the non-padding tokens and feed the pooled vector into
9 small classification heads, one per aspect. Each head emits a
4-way softmax over {absent, positive, negative, neutral}. This frames
presence-detection and sentiment together as a single choice, which
saves us from the classic mismatch where aspect detection fires but
sentiment hedges.
The input isn't just the raw review — we prepend
[platform|category|Nstar]. Short reviews ⇒ any cheap prior helps,
and platform + category are strong signals on their own (play_store
implies app_experience, restaurant implies food).
Loss is inverse-frequency weighted cross-entropy per (aspect, state)
bin — the dataset is heavily skewed (service is half the rows, none is
3%, neutral is 4.5% of all sentiments) so without weights the model just
predicts service.
Training config that mattered: AdamW with 2e-5 on the backbone and 1e-4 on the heads, 10% warmup then linear decay, gradient clip at 1.0, multi-sample dropout (4 samples averaged) on the runs trained with pseudo-labels.
pip install gradio
python app.py
Auto-picks the latest checkpoint; paste a review, hit Analyze.
multilingual.py
covered the common cases.none on
single-word greetings, force app_experience=negative on play_store
MIT on the code. Weights inherit MARBERTv2's upstream terms. "# NLP"