query-context-pruner-multilingual

A multilingual model designed to remove contexts irrelevant to queries. Since this model is a relatively large LLM, practical pruning methods like Provence are needed for speed-critical information retrieval tasks. This model is particularly well-suited for generating teacher labels for training such efficient pruning models.

We offer two model sizes: 4B model for accuracy-focused label generation where precision matters more than speed, and 1.7B model for high-speed label generation.

Why This Model is Valuable

Based on the Provence paper, context pruning addresses critical challenges in RAG systems:

• Context Noise Problem: Retrieved documents often contain irrelevant sentences that mislead LLMs and degrade response quality
• Computational Overhead: Long contexts slow down generation and increase costs significantly
• Performance Degradation: Irrelevant information leads to hallucinations and poor answer quality

Our model enables efficient training data generation for lightweight context pruners that can:

• Reduce context length by 70-90% while maintaining accuracy
• Achieve almost zero-cost pruning when unified with reranking (as shown in Provence)
• Work out-of-the-box across diverse domains and languages
• Dynamically detect optimal pruning ratios (0-100%) per context

This addresses the practical need for robust, adaptable context pruners that can be deployed in production RAG systems without compromising performance.

🚀 Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Choose model size
MODEL_NAME = "hotchpotch/query-context-pruner-multilingual-Qwen3-4B"   # Recommended (best balance)
# MODEL_NAME = "hotchpotch/query-context-pruner-multilingual-Qwen3-1.7B"  # Faster alternative

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)


def create_user_prompt(query: str, contexts: list[str]) -> str:
    """Create a formatted prompt from query and context chunks."""
    context_str = "\n".join([f"[{i+1}] {ctx}" for i, ctx in enumerate(contexts)])
    return f"{query}\n---\n{context_str}"


# Example: Python documentation split into chunks
query = "How do you read and write files in Python?"
contexts = [
    "Python is a high-level programming language known for its simplicity and readability.",
    "To read a file in Python, you use the open() function with the 'r' mode. For example: with open('file.txt', 'r') as f: content = f.read()",
    "Writing to files in Python also uses open() with 'w' mode for writing or 'a' mode for appending. Example: with open('file.txt', 'w') as f: f.write('Hello')",
    "Python supports multiple programming paradigms including object-oriented and functional programming.",
    "The 'with' statement ensures proper file handling by automatically closing files after use, preventing resource leaks.",
    "Python has various built-in functions like len(), range(), and type() that are commonly used in everyday programming."
]

# Generate response
prompt = create_user_prompt(query, contexts)
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=128, temperature=0.1, do_sample=True)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)

print(f"Relevant chunks: {response}")  # Expected: "2,3,5" (reading, writing, and proper file handling)

# Note: The model's output indices start from 1, not 0

📋 Model Description

Query-Context Pruner is designed to solve the context overload problem in RAG systems by identifying which text chunks contain information relevant to answering a query. This enables:

• Efficient RAG pipelines: Reduce context length by 70-90% while maintaining accuracy
• Training data generation: Create high-quality labels for smaller bi-encoder models
• Multilingual information retrieval: Works across 20 languages with varying performance
• Context compression: Eliminate irrelevant information before passing to LLMs

🌍 Supported Languages

The model supports 20 languages with varying performance levels. Important: While some languages show excellent performance, others have limited effectiveness. Please carefully review your target language's performance before use.

Note on Evaluation: Many test datasets (especially MIRACL) have very small sample sizes, making test scores less reliable. We report averaged F1 scores (test + validation) for more robust evaluation.

Language Performance Summary (4B Model)

* Average across multiple datasets (Japanese: 8 datasets, English: MS-MARCO + MIRACL)
† Very small test set - interpret with caution
‡ High variance between test/validation - less stable performance

Note: Performance varies by domain. Languages with more training data generally perform better.

📊 Model Evaluation

We evaluated four model sizes on 28 datasets covering multiple languages and domains. Results shown are combined test+validation scores for robustness.

Model Size Comparison

Performance Highlights

Why 4B Model is Recommended:

• Only 0.9% F1 gap with 8B model (0.820 vs 0.827)
• 67% faster inference than 8B model
• Perfect balance of accuracy and efficiency
• Optimal for production deployments

Detailed Performance by Dataset (4B Model)

Note: We show both test and validation scores because some datasets have small test sets, making validation scores more reliable. Large differences between splits (>0.1) may indicate data quality issues.

† Important: All MIRACL datasets have fewer than 100 test samples, with many having fewer than 50. During dataset creation, we should have allocated more samples to the test sets for MIRACL languages. The small sample sizes raise concerns about the statistical reliability of the reported metrics. Performance on these datasets should be interpreted with caution, as the confidence intervals are likely to be wide.

🎯 Model Selection Guide

Choose 4B (Recommended) if you need:

• Optimal balance of accuracy and speed
• Production deployment
• F1 > 0.80 on major languages
• Cost-effective inference
• Best performance per computational cost

Choose 1.7B if you need:

• Maximum speed for real-time applications
• Resource-constrained environments
• F1 > 0.75 is sufficient
• Edge deployment scenarios

📚 Training Details

• Dataset: qa-context-relevance-multilingual-140k
  • 142,629 samples across 20 languages
  • English: MS-MARCO dataset for robust English QA performance
  • Japanese: Various Japanese QA datasets (JaQuAD, JSQuAD, JQARA, Quiz datasets) for comprehensive coverage
  • Multilingual: MIRACL dataset covering 18 languages for broad multilingual support
  • Important Note: Relevance labels were generated using DeepSeek-V3-0324, an LLM, without human verification. Therefore, the ground truth data itself may contain errors and should not be considered perfect annotations.
• Base Model: Qwen-3 series
• Method: Supervised Fine-Tuning (SFT)
• Training Strategy: Multi-language curriculum learning

🔧 Technical Implementation

The model implements context pruning as described in:

Provence: Efficient and Robust Context Pruning for Retrieval-Augmented Generation
Zhang et al., 2025 (arXiv:2501.16214)

Key features:

• Citation-style output format (e.g., "1,3" for chunks 1 and 3)
• Handles multiple relevant chunks
• Language-agnostic prompt format
• Optimized for chunk-level relevance

📄 License

Apache-2.0

👤 Author

Yuichi Tateno (@hotchpotch)

---

📖 Appendix

A. Dataset Creation Process

The training dataset was created through:

1. Collection of diverse QA datasets:
   • MS-MARCO for English
   • Japanese QA datasets (JaQuAD, JSQuAD, JQARA, MKQA-ja, Mr.TyDi-ja, Quiz-no-mori, Quiz-works)
   • MIRACL for 18 additional languages
2. Text chunking using language-specific tools
3. Relevance annotation using DeepSeek-V3-0324 (LLM-based annotation)
4. Quality filtering and balancing

Note: All relevance annotations were generated by an LLM without human verification, which may introduce annotation errors.

B. Model Architecture

• Base: Qwen-3 transformer architecture
• Context Length: Standard context window
• Special Tokens: Standard Qwen-3 chat template
• Output Format: Citation numbers or "No answer"

C. Evaluation Methodology

We evaluated on both test and validation sets because:

• Some datasets have small test sets (<100 samples)
• Validation scores provide more stable estimates
• Combined scores reduce evaluation noise
• Split differences help identify data quality issues

D. Performance Comparison Across Model Sizes

D.1 Japanese Datasets Performance

D.2 English Datasets Performance

D.3 MIRACL Languages Performance (Selected High/Low Performers)

Key Observations:

1. Diminishing Returns: The performance gain from 4B to 8B is minimal (avg 1-3% improvement) while computational cost increases significantly
2. Language-Specific Patterns: Some languages (e.g., Telugu, Hindi) show poor performance across all model sizes, indicating data quality issues
3. 4B Sweet Spot: The 4B model achieves 95-98% of 8B performance at 60% of the computational cost
4. Small Sample Impact: Languages with <50 test samples show high variance between test/validation scores