Russian Promptriever

A monolingually-trained Russian instruction-following dense retriever, built by replicating the Promptriever recipe (Weller et al., 2024) on Qwen3-4B-Base with LoRA fine-tuning. HSE FCS DSBA bachelor's thesis, defense June 2026.

Headline Results

Evaluated on mFollowIR-Russian (n = 40 paired query-instruction pairs):

The trained model is statistically indistinguishable from mE5-large and BGE-M3 on p-MRR (Holm-corrected Wilcoxon p ≥ 0.92, n = 40); the lift in retrieval quality over the same-backbone zero-shot baseline is significant. A cross-lingual evaluation observation: the English-trained Promptriever-Llama-2 attains p-MRR = +0.0362 on the same Russian harness without any Russian-language training data, suggesting that instruction-following transfers cross-lingually while the retrieval substrate must be built in-language.

Full numeric outputs of the evaluation campaign are in step7/results/.

Pipeline

Saved Server Outputs (evidence)

Outputs from each stage that survive in the repo:

• step5/scores_qwen3/, step5/scores_bge/ — Step-5 cross-encoder scoring outputs (parquet) from the dual BGE + Qwen3-Reranker pass on 489,243 candidate rows.
• step5/results_qwen3/dropped_qids.json — canonical drop list (73,387 query IDs at the paper-parity 15.0 % drop rate).
• step7/results/ — full mFollowIR-Russian leaderboard outputs, per-query persistence JSONs, Holm-corrected Wilcoxon outputs for the four planned baselines, ruMTEB triangulation outputs, cross-lingual inversion outputs (Promptriever-Llama-2 vs. ours), instruction kill-switch ablation, LoRA α-rescale grid, and the RRF ensemble run.
• evaluation/bge_validation/ — BGE-reranker validation harness.
• assembly/token_length_stats.json, assembly/validation_stats.json — corpus audit summaries (token-length distribution and the I1–I9 invariant validator output).

Heavy intermediate data (raw mMARCO collection, OpenAI Batch API outputs, model checkpoints, optimizer states) is regenerable from the scripts and is not committed; see .gitignore for the gitignored set.

Trained Adapter

A single LoRA adapter (Qwen3-4B-Base + LoRA r=32, α=64, rank-stabilised scaling, all seven projection matrices) was trained for 7,085 steps (one contrastive epoch on 906,863 rows) over approximately 11 hours on 4× H100 NVL devices (94 GB HBM3 each), with zero OOM and zero NaN events along the run.

Adapter weights (252 MB safetensors) are too large for GitHub. To be published on the Hugging Face Hub on release.

Reproducibility

• training/scripts/train.py — YAML-driven torchrun shim for the main training run; references the local Tevatron patch in training/patches/ (adds the use_rslora flag).
• training/configs/qwen3_4b.yaml — single source of truth for hyperparameters used in the main run.
• step5/score_pairs_qwen3_vllm.py, step5/launch_4gpu_vllm.sh — Step-5 vLLM-based cross-encoder pass.
• step7/ — evaluation campaign code, per-query persistence patch for mteb 1.18.4, paired-Wilcoxon + BCa bootstrap routines.

Dependencies

• Python 3.11
• PyTorch with CUDA 12.x, FlashAttention 2, Liger Kernel
• HuggingFace transformers, peft, datasets
• Tevatron with the local use_rslora patch (training/patches/)
• mteb 1.18.4 with the per-query persistence patch (step7/)
• pyarrow, ftfy, pymorphy3
• OpenAI Python SDK (for the translation pipeline)
• vLLM 0.21+ (for the Step-5 cross-encoder scoring pass)

License

Code and configurations are released under the Apache 2.0 License.

The Russian PromptrieverDocs corpus inherits the licensing of its English source samaya-ai/msmarco-w-instructions and of the Russian mMARCO collection (Bonifacio et al., 2021).