PrestoAI Arabic E-Commerce Search & Retrieval Challenge
Team Retrieval Renegades · Marwan Elamami · Kaggle marwanelamami13
Public leaderboard nDCG@10 = 0.64335 — rank 3 at time of writing, up from 0.57424 over eleven days and nineteen logged submissions.
A measurement-driven hybrid ranking system for Arabic product search: six retrieval arms fused by weighted Reciprocal Rank Fusion, re-ranked by a three-model ensemble built around a zero-shot Qwen3-Reranker-0.6B used as a CausalLM yes/no judge, blended with the fusion signal under per-query z-normalisation. Every weight, constant and depth in the system was chosen by a logged public-leaderboard A/B test — never by offline validation.
📄 docs/paper/main.pdf — 27 pages, 5 diagrams, full method and evidence
📊 results/experiments.csv — every submission, config and score
Queries are normalised (alif/ya/ta-marbuta unification, diacritics stripped, case folded).
Six arms each rank the 107,041-product catalog: BM25 over a stemmed view, BM25 over an
unstemmed view, character 3–6-gram TF-IDF, a two-stage contrastively fine-tuned
Qwen3-Embedding-0.6B bi-encoder (with one round of pseudo-relevance feedback on the query
vectors), Meilisearch, and GATE-AraBERT. Arms are fused by weighted RRF with k = 15 —
deliberately sharper than the conventional 60, because the metric truncates at ten. The
fused top-100 is scored by three re-rankers: two fine-tuned bge-reranker-v2-m3
checkpoints and a zero-shot Qwen3-Reranker-0.6B at double weight. Their per-query
z-scores are averaged and blended with the z-scored fusion signal at w = 0.35 — the
re-rankers hold a deliberate minority vote — and the top ten is emitted under a stable
sort, so ties fall back to fusion order.
Document views are not uniform across arms, and that is intentional: §6.1 of the paper states which arms see normalised text, which see raw text, and why.
| Finding | Evidence |
|---|---|
| Supervised fine-tuning of re-rankers hurts here. Pointwise training on the provided synthetic labels raises held-out pairwise accuracy by 12 points while degrading leaderboard nDCG. | Three independently trained models: r2 (mined negatives, 0.63757), v3 (2.6× data, no effect), LoRA-adapted qwen (0.63653 vs 0.63972 zero-shot) |
| Ensemble value comes from decorrelation, not strength. | The generative judge correlates +0.42 with the cross-encoders and earned double weight; a better-trained listwise cross-encoder correlating +0.75 contributed +0.00004 |
| Ordering, not retrieval, was the bottleneck. 90.8% of relevant products were already in the fused pool; only 56.1% reached the scored top ten. | Cancelled a seventh arm and a dense retrain on the spot; redirected everything to re-ranking |
| Re-ranking depth is an untuned axis. Extending the re-ranked prefix from the conventional 50 to 100 produced the final champion. | 0.64315 → 0.64335 |
The largest single gain in the project was +0.0298, from adding a differently-wrong retriever (character n-gram TF-IDF) — not from improving an existing one. The largest single effect was −0.044, from deduplicating results: the judges graded identifiers, not products, so removing a duplicate deletes a correct answer.
pip install -e ".[dev]" # installs the presto package + pytest
pytest # 40 tests, ~1 s, no GPU / data / network
Rebuild the champion submission from cached exports (~2 min, CPU):
python -m presto.build_submission --exports <exports-dir> --out submission.csv
make help lists the other entry points.
src/presto/ Installable, unit-tested library (the reference system)
text.py normalize_arabic, rich_product_text, format_query
metrics.py dcg, evaluate (graded nDCG@k + recall / MRR)
ranking.py weighted_rrf, reranker_consensus, blend, rank_one_query
build_submission.py CLI: rebuild the champion submission from exports
pipelines/ Standalone Kaggle GPU kernels (Stage I + III) — see pipelines/README.md
retrieve_fuse_rerank.py THE champion kernel: 6 arms, weighted RRF,
zero-shot Qwen3-Reranker at depth 100
retrieve_fuse_rerank_earlier.py Earlier sprint version (pre-rebuild)
retrieve_fuse_rerank_listwise.py Variant scoring the listwise cross-encoder
score_listwise_cross_encoder.py Standalone listwise scoring pass
training/ One file per training run — see training/README.md
bi_encoder.py Bi-encoder, two-stage contrastive, single 16 GB T4
bi_encoder_lora.py LoRA variant; PyArabic + CAMeL Tools normalisation
cross_encoder_r1_pointwise.py r1 — ADOPTED (+0.0032)
cross_encoder_r2_mined_negatives.py r2 — HARMFUL (0.63757)
cross_encoder_v3_scaled_data.py v3, 2.6× supervision — NO EFFECT
cross_encoder_v4_listwise.py v4, listwise LambdaLoss — +0.00004
evaluation/ Component benchmarks — see evaluation/README.md
benchmark_bi_encoder.py Standalone benchmark of the dense arm
ab_test_bi_encoders.py Bi-encoder A/B harness
experiments/ Exploration + tools — see experiments/README.md
cpu_blend_factory.py The CPU variant factory over cached exports
try_*.py Hypotheses tested and logged
merge_*.py Fold an external score export into the factory
notebooks/ Colab runners (self-contained) — see notebooks/README.md
docs/
paper/main.tex / .pdf 27 pp. Self-contained; XeLaTeX (Arabic set in Amiri).
kaggle_kernels.md In-kernel pipeline documentation
tests/ 40 unit tests: ranking arithmetic, Arabic
normalisation, nDCG. Expectations are computed by
hand in the tests, not copied from runs.
results/
experiments.csv Complete experiment ledger
submission_log.md Curated leaderboard history with conclusions
Every code folder carries a README.md mapping its files.
Naming conventions. Under pipelines/, the bare name is the canonical
champion and a suffix marks a variant. Under experiments/, try_* is a
logged hypothesis and merge_* folds in an external score export. Under
training/, the r1/r2/v3/v4 labels are the paper's own taxonomy for the
four cross-encoder rounds.
Reading order. The four training/cross_encoder_* files are the empirical spine of this
work and are best read in sequence — each header states its hypothesis, the evidence that
motivated it, and what the leaderboard said. Together they document why supervised
fine-tuning on this dataset's labels improves accuracy while degrading ranking.
Everything expensive runs once. Indexing, embedding and re-ranker scoring write Parquet; every ranking decision above that layer is arithmetic over cached arrays. A new configuration costs ~2 minutes of laptop CPU instead of hours of GPU time, which is what made one-variable-per-submission practical across 30+ logged experiments.
Fail closed, never plausibly. The pipeline spans heterogeneous environments and re-joins outputs afterwards, so its characteristic failure is a misalignment — a submission that is syntactically perfect and silently wrong. Every merge asserts exact identifier parity (29,014 of 29,014 rows) and aborts otherwise. The same reasoning gives the gated checkpoint promotion and the finite-score assertions.
Numerics on a T4. Qwen3 is bf16-native and Turing exposes no bf16 path, so fp16 autocast overflows. Training runs in fp32 with a non-finite-loss counter; inference uses fp16 autocast with automatic per-batch fp32 recomputation whenever any output is non-finite.
One bug the tests found. zscore guarded a constant prefix with v.std() or 1.0,
which only catches a standard deviation of exactly zero. A prefix equal up to
floating-point error has std ≈ 1e-16, and dividing by it amplified pure rounding noise into
a full-scale signal that reordered tied candidates arbitrarily. Now guarded with a
tolerance — see tests/test_ranking.py::test_zscore_of_constant_vector_is_zero_not_nan.
The public benchmark (prestoai/arabic-ecom-search-bench) coincides with the hidden test
set — scoring the champion submission against the local qrels.tsv reproduces the
leaderboard to within 0.0008. From 2026-07-19 onward we therefore enforced a strict
protocol:
qrels.tsv or candidates.jsonl.results/experiments.csv maps every score to
a configuration.Early exploratory work predating this protocol was discarded and is not part of any claimed score; §4.2 of the paper documents the transition, including the withdrawal of a higher label-assisted result that the rebuilt pipeline surpassed honestly three days later.
Models are trained exclusively on the provided training data (prestoai/arabic-ecom-data).
No Hugging Face token is committed anywhere in this repository; scripts read HF_TOKEN
from the environment.
MIT — see LICENSE.
PrestoAI Arabic E-Commerce Search & Retrieval Challenge
Team Retrieval Renegades · Marwan Elamami · Kaggle marwanelamami13
Public leaderboard nDCG@10 = 0.64335 — rank 3 at time of writing, up from 0.57424 over eleven days and nineteen logged submissions.
A measurement-driven hybrid ranking system for Arabic product search: six retrieval arms fused by weighted Reciprocal Rank Fusion, re-ranked by a three-model ensemble built around a zero-shot Qwen3-Reranker-0.6B used as a CausalLM yes/no judge, blended with the fusion signal under per-query z-normalisation. Every weight, constant and depth in the system was chosen by a logged public-leaderboard A/B test — never by offline validation.
📄 docs/paper/main.pdf — 27 pages, 5 diagrams, full method and evidence
📊 results/experiments.csv — every submission, config and score
Queries are normalised (alif/ya/ta-marbuta unification, diacritics stripped, case folded).
Six arms each rank the 107,041-product catalog: BM25 over a stemmed view, BM25 over an
unstemmed view, character 3–6-gram TF-IDF, a two-stage contrastively fine-tuned
Qwen3-Embedding-0.6B bi-encoder (with one round of pseudo-relevance feedback on the query
vectors), Meilisearch, and GATE-AraBERT. Arms are fused by weighted RRF with k = 15 —
deliberately sharper than the conventional 60, because the metric truncates at ten. The
fused top-100 is scored by three re-rankers: two fine-tuned bge-reranker-v2-m3
checkpoints and a zero-shot Qwen3-Reranker-0.6B at double weight. Their per-query
z-scores are averaged and blended with the z-scored fusion signal at w = 0.35 — the
re-rankers hold a deliberate minority vote — and the top ten is emitted under a stable
sort, so ties fall back to fusion order.
Document views are not uniform across arms, and that is intentional: §6.1 of the paper states which arms see normalised text, which see raw text, and why.
| Finding | Evidence |
|---|---|
| Supervised fine-tuning of re-rankers hurts here. Pointwise training on the provided synthetic labels raises held-out pairwise accuracy by 12 points while degrading leaderboard nDCG. | Three independently trained models: r2 (mined negatives, 0.63757), v3 (2.6× data, no effect), LoRA-adapted qwen (0.63653 vs 0.63972 zero-shot) |
| Ensemble value comes from decorrelation, not strength. | The generative judge correlates +0.42 with the cross-encoders and earned double weight; a better-trained listwise cross-encoder correlating +0.75 contributed +0.00004 |
| Ordering, not retrieval, was the bottleneck. 90.8% of relevant products were already in the fused pool; only 56.1% reached the scored top ten. | Cancelled a seventh arm and a dense retrain on the spot; redirected everything to re-ranking |
| Re-ranking depth is an untuned axis. Extending the re-ranked prefix from the conventional 50 to 100 produced the final champion. | 0.64315 → 0.64335 |
The largest single gain in the project was +0.0298, from adding a differently-wrong retriever (character n-gram TF-IDF) — not from improving an existing one. The largest single effect was −0.044, from deduplicating results: the judges graded identifiers, not products, so removing a duplicate deletes a correct answer.
pip install -e ".[dev]" # installs the presto package + pytest
pytest # 40 tests, ~1 s, no GPU / data / network
Rebuild the champion submission from cached exports (~2 min, CPU):
python -m presto.build_submission --exports <exports-dir> --out submission.csv
make help lists the other entry points.
src/presto/ Installable, unit-tested library (the reference system)
text.py normalize_arabic, rich_product_text, format_query
metrics.py dcg, evaluate (graded nDCG@k + recall / MRR)
ranking.py weighted_rrf, reranker_consensus, blend, rank_one_query
build_submission.py CLI: rebuild the champion submission from exports
pipelines/ Standalone Kaggle GPU kernels (Stage I + III) — see pipelines/README.md
retrieve_fuse_rerank.py THE champion kernel: 6 arms, weighted RRF,
zero-shot Qwen3-Reranker at depth 100
retrieve_fuse_rerank_earlier.py Earlier sprint version (pre-rebuild)
retrieve_fuse_rerank_listwise.py Variant scoring the listwise cross-encoder
score_listwise_cross_encoder.py Standalone listwise scoring pass
training/ One file per training run — see training/README.md
bi_encoder.py Bi-encoder, two-stage contrastive, single 16 GB T4
bi_encoder_lora.py LoRA variant; PyArabic + CAMeL Tools normalisation
cross_encoder_r1_pointwise.py r1 — ADOPTED (+0.0032)
cross_encoder_r2_mined_negatives.py r2 — HARMFUL (0.63757)
cross_encoder_v3_scaled_data.py v3, 2.6× supervision — NO EFFECT
cross_encoder_v4_listwise.py v4, listwise LambdaLoss — +0.00004
evaluation/ Component benchmarks — see evaluation/README.md
benchmark_bi_encoder.py Standalone benchmark of the dense arm
ab_test_bi_encoders.py Bi-encoder A/B harness
experiments/ Exploration + tools — see experiments/README.md
cpu_blend_factory.py The CPU variant factory over cached exports
try_*.py Hypotheses tested and logged
merge_*.py Fold an external score export into the factory
notebooks/ Colab runners (self-contained) — see notebooks/README.md
docs/
paper/main.tex / .pdf 27 pp. Self-contained; XeLaTeX (Arabic set in Amiri).
kaggle_kernels.md In-kernel pipeline documentation
tests/ 40 unit tests: ranking arithmetic, Arabic
normalisation, nDCG. Expectations are computed by
hand in the tests, not copied from runs.
results/
experiments.csv Complete experiment ledger
submission_log.md Curated leaderboard history with conclusions
Every code folder carries a README.md mapping its files.
Naming conventions. Under pipelines/, the bare name is the canonical
champion and a suffix marks a variant. Under experiments/, try_* is a
logged hypothesis and merge_* folds in an external score export. Under
training/, the r1/r2/v3/v4 labels are the paper's own taxonomy for the
four cross-encoder rounds.
Reading order. The four training/cross_encoder_* files are the empirical spine of this
work and are best read in sequence — each header states its hypothesis, the evidence that
motivated it, and what the leaderboard said. Together they document why supervised
fine-tuning on this dataset's labels improves accuracy while degrading ranking.
Everything expensive runs once. Indexing, embedding and re-ranker scoring write Parquet; every ranking decision above that layer is arithmetic over cached arrays. A new configuration costs ~2 minutes of laptop CPU instead of hours of GPU time, which is what made one-variable-per-submission practical across 30+ logged experiments.
Fail closed, never plausibly. The pipeline spans heterogeneous environments and re-joins outputs afterwards, so its characteristic failure is a misalignment — a submission that is syntactically perfect and silently wrong. Every merge asserts exact identifier parity (29,014 of 29,014 rows) and aborts otherwise. The same reasoning gives the gated checkpoint promotion and the finite-score assertions.
Numerics on a T4. Qwen3 is bf16-native and Turing exposes no bf16 path, so fp16 autocast overflows. Training runs in fp32 with a non-finite-loss counter; inference uses fp16 autocast with automatic per-batch fp32 recomputation whenever any output is non-finite.
One bug the tests found. zscore guarded a constant prefix with v.std() or 1.0,
which only catches a standard deviation of exactly zero. A prefix equal up to
floating-point error has std ≈ 1e-16, and dividing by it amplified pure rounding noise into
a full-scale signal that reordered tied candidates arbitrarily. Now guarded with a
tolerance — see tests/test_ranking.py::test_zscore_of_constant_vector_is_zero_not_nan.
The public benchmark (prestoai/arabic-ecom-search-bench) coincides with the hidden test
set — scoring the champion submission against the local qrels.tsv reproduces the
leaderboard to within 0.0008. From 2026-07-19 onward we therefore enforced a strict
protocol:
qrels.tsv or candidates.jsonl.results/experiments.csv maps every score to
a configuration.Early exploratory work predating this protocol was discarded and is not part of any claimed score; §4.2 of the paper documents the transition, including the withdrawal of a higher label-assisted result that the rebuilt pipeline surpassed honestly three days later.
Models are trained exclusively on the provided training data (prestoai/arabic-ecom-data).
No Hugging Face token is committed anywhere in this repository; scripts read HF_TOKEN
from the environment.
MIT — see LICENSE.