ru-Promptriever-Qwen3-4B

---

Overview

Standard dense retrieval models score query–passage pairs using a single semantic similarity signal, giving users no control over what "relevant" means beyond keyword choice. Promptriever (Weller et al., 2024) introduced per-instance natural language instructions that dynamically redefine relevance — a capability previously limited to generative LLMs.

ru-Promptriever extends this paradigm to Russian:

• Architecture: Qwen3-based causal LM fine-tuned as a bi-encoder with LoRA + GradCache
• Pooling: last-token (EOS) pooling
• Key training signal: instruction negatives — passages that are topically relevant to the query but violate the instruction constraint

Model Family

---

This Model

This is the final, best-performing model in the ru-Promptriever family. Starting from ru-Promptriever-4B-pretrained, it was further fine-tuned on a balanced multilingual mix of:

• Russian real data — MIRACL + MrTyDi (~11k pairs)
• Russian synthetic data — instruction-augmented and standard pairs (~30k)
• English instruction-following data from the original Promptriever / FollowIR dataset (~24k, ~35% of mix)

Under the current harmonized evaluation protocol, this model obtains 17.28 p-MRR on mFollowIR-RU while matching Promptriever-8B's retrieval quality at half the parameter count.

---

Evaluation Results

mFollowIR-RU

Russian split of mFollowIR — multilingual instruction-following retrieval using TREC NeuCLIR narratives as instructions.

p-MRR (Pairwise Mean Reciprocal Rank, ×100) is the primary instruction-following metric — higher means the model correctly adjusts rankings when instructions change. nDCG@20 measures standard retrieval quality.

Both models were rerun on the official MTEB 2.10.5 candidate pools with their native preprocessing. The paired nDCG@20 difference is +0.0003 (95% CI [−0.047, +0.049], p=0.991). The p-MRR difference is numerically positive but not statistically significant (95% CI [−1.36, +11.57], p=0.156). These results support higher observed instruction sensitivity without an established retrieval-quality loss, not proven overall superiority.

Additional Benchmarks

These values use the repository's selected step-300 adapter and the corrected MTEB 2.10.5 protocol. FollowIR retrieval columns use the original-instruction qrels; legacy published retrieval values from older FollowIR implementations are not directly comparable.

---

Usage

Basic Retrieval (no instruction)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch.nn.functional as F

model_name = "Vladimirlv/ru-promptriever-qwen3-4b"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

def encode(texts: list[str], max_length: int = 512) -> torch.Tensor:
    """Encode texts using last-token (EOS) pooling."""
    inputs = tokenizer(
        texts,
        padding=True,
        truncation=True,
        max_length=max_length,
        return_tensors="pt",
    ).to(model.device)

    with torch.no_grad():
        # Bypass lm_head to get post-norm hidden states
        original_lm_head = model.lm_head
        model.lm_head = torch.nn.Identity()
        outputs = model(**inputs, use_cache=False, return_dict=True)
        model.lm_head = original_lm_head

    # EOS pooling: take embedding at last non-padding token
    seq_len = inputs["attention_mask"].sum(dim=1) - 1
    embeddings = outputs.logits[torch.arange(len(texts)), seq_len]
    return F.normalize(embeddings, p=2, dim=1)


query = "Когда была основана Москва?"
passages = [
    "Москва была основана в 1147 году князем Юрием Долгоруким.",
    "Санкт-Петербург был основан Петром I в 1703 году.",
]

q_emb = encode([query])
p_emb = encode(passages)
scores = (q_emb @ p_emb.T).squeeze()
print(scores)  # tensor([0.82, 0.61])

Instruction-Following Retrieval

# Append the instruction directly to the query (same format as training)
instruction = "Найди документ, в котором упоминается конкретная дата основания города."
instructed_query = f"{query} {instruction}"

q_emb = encode([instructed_query])
p_emb = encode(passages)
scores = (q_emb @ p_emb.T).squeeze()
# The model adjusts rankings based on the instruction

Using with sentence-transformers

This model is not compatible with sentence-transformers out of the box due to the custom EOS pooling. Use the snippet above directly with transformers.

---

Model Details

---

Training Data

This model was trained in two stages:

Stage 1 — Pretraining (ru-Promptriever-4B-pretrained): LoRA fine-tuning of Qwen3-4B on ~500k synthetic instruction-augmented Russian retrieval triples from ru-promptriever-dataset, built on top of mMARCO-ru.

Stage 2 — Continued training (this model): further LoRA fine-tuning on a balanced multilingual mix of ~65k rows:

1. Russian real data (~11k) — from MIRACL and MrTyDi retrieval datasets
2. Russian synthetic data (~30k) — instruction-augmented and standard pairs from ru-promptriever-dataset
3. English synthetic data (~24k, ~35% of total) — from the original Promptriever / FollowIR training data

Key properties:

• Instruction negatives: passages that are topically relevant but violate the instruction (3 per instructed query)
• Paired rows: each source query has both a standard row and an instructed row to prevent catastrophic forgetting

---

Intended Use

• Instruction-following dense retrieval in Russian: RAG pipelines, search systems, and scenarios requiring fine-grained query control via natural language
• Research on multilingual instruction-following retrieval and bi-encoder training
• Benchmarking alongside mE5, BGE-M3, and Promptriever-style models

Out-of-Scope

• General-purpose text embedding (use mE5-large or BGE-M3 if no instruction-following is needed)
• Commercial applications (see License below)

---

Limitations

• MS MARCO origin: The training corpus derives from English web passages machine-translated to Russian. A portion of passages retain translation artifacts despite LLM-based rewriting.
• Standard retrieval trade-off: Instruction-following training slightly reduces standard retrieval quality compared to encoder-only models (mE5-large, BGE-M3).
• Synthetic-data quality: A blinded two-author audit of 64 final instances found 95.2% positive-document compliance and 85.2% negative non-compliance with the complete query-instruction condition (79.3% agreement; Cohen's κ = 0.57). The audit is encouraging but small, and imperfect synthetic examples may remain.

---

License

This model is released under CC BY-NC 4.0 (Creative Commons Attribution–NonCommercial 4.0 International).

The non-commercial restriction is inherited from the upstream MS MARCO license (Microsoft Research License — non-commercial use only), which governs the training corpus.

---

Citation

If you use this model, please cite the original Promptriever paper:

@article{weller2024promptriever,
  title   = {Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models},
  author  = {Weller, Orion and Van Durme, Benjamin and Lawrie, Dawn and
             Paranjape, Ashwin and Zhang, Yuhao and Hessel, Jack},
  journal = {arXiv preprint arXiv:2409.11136},
  year    = {2024}
}