---

📢 Latest News

---

🔥 Latest Papers

---

🌳 Table of Contents

• Latest News
• Latest Papers
• About This Repository
• Overview
• Key Insights from the Survey
• Paper List
  • 1. Foundational LLM-as-a-Judge Work
  • 2. Specialized & Fine-Tuned Judge Models
  • 3. Token Budget, Reasoning Efficiency & Overthinking
  • 4. Ensemble & Panel-Based Methods
  • 5. Multi-Agent Debate & Deliberation
  • 6. Persona Effects & Prompt Sensitivity
  • 7. Judge Biases, Preference Leakage & Uncertainty
  • 8. Adversarial Robustness
  • 9. Multilingual Evaluation
  • 10. Self-Evolving & Self-Improving Judges
  • 11. Alignment, RLHF & Training Methods
  • 12. Reasoning & Chain-of-Thought
  • 13. Model Papers
• Benchmarks & Datasets
• Related Surveys
• Citation
• Contributing

---

🌟 About This Repository

This repository accompanies our survey paper "Small Language Models as Judges: A Survey", which provides the first comprehensive review of the emerging paradigm where Small Language Models (SLMs), operationally defined as models with ≤14B parameters that can run on a single consumer GPU, serve as automated evaluators.

We organize the rapidly expanding literature around a structured taxonomy spanning five key dimensions:

1. Background & Motivation: Why SLM judges, formal definitions, and evaluation formalism
2. Individual SLM Judging: Specialized judges, token budgets, capability thresholds, overthinking, biases, and thinking modes
3. Multi-Agent Strategies: Ensembles, debate, output optimization, and persona robustness
4. Benchmarks & Metrics: Meta-evaluation infrastructure, adversarial robustness
5. Challenges & Future Directions: Open research problems along the verifiability gradient

We will continuously update this repository with the latest papers and resources. If you find this helpful, please ⭐ star the repo!

📬 Missing a paper? Feel free to open an issue or submit a pull request.

---

📊 Overview

Figure: Taxonomy of the SLM-as-a-Judge survey, organized around five key dimensions: background and motivation, individual SLM judging, multi-agent strategies, benchmarks and metrics, and challenges and future directions.

---

🔬 Key Insights from the Survey

Our survey identifies five convergent insights with multi-paper evidence:

Left: Scale vs. capability, where fine-tuned 3-8B judges match proprietary models on standard benchmarks. Right: Reasoning tokens help only when they carry new discriminative signal.

Left: Heterogeneous SLM juries decorrelate errors more effectively than scaling a single judge. Right: The verifiability gradient, where SLM judges approach parity in high-verifiability domains but degrade for subjective tasks.

The cost-quality frontier: SLM judges occupy a practical sweet spot for most automated evaluation.

---

📑 Paper List

1. Foundational LLM-as-a-Judge Work

---

2. Specialized & Fine-Tuned Judge Models

Figure: Comparison of training paradigms for SLM judge systems.

---

3. Token Budget, Reasoning Efficiency & Overthinking

---

4. Ensemble & Panel-Based Methods

Figure: Taxonomy of multi-agent evaluation strategies: ensembles, juries, and debate.

---

5. Multi-Agent Debate & Deliberation

Figure: Failure modes in multi-agent debate systems (MAST taxonomy).

---

6. Persona Effects & Prompt Sensitivity

---

7. Judge Biases, Preference Leakage & Uncertainty

Figure: Heatmap of structural biases across judge models and evaluation settings.

---

8. Adversarial Robustness

Figure: Defense strategies against adversarial attacks on LLM judges.

---

9. Multilingual Evaluation

---

10. Self-Evolving & Self-Improving Judges

---

11. Alignment, RLHF & Training Methods

Figure: Supervision paradigms for training SLM judges.

---

12. Reasoning & Chain-of-Thought

---

13. Model Papers

Figure: Capability heatmap across SLM model families.

---

📦 Benchmarks & Datasets

Meta-Evaluation Benchmarks

Figure: Coverage of meta-evaluation benchmarks across evaluation dimensions.

Reasoning & Math Benchmarks (Used as Evaluation Substrates)

Code & Safety Benchmarks

Foundational NLP Evaluation Metrics

---

📚 Related Surveys

---

📖 Citation

If you find this survey and repository useful for your research, please cite our paper:

@article{laddha2026slmjudge,
  title   = {Small Language Models as Judges: A Survey},
  author  = {Anish Laddha and Nitesh Pradhan and Gaurav Srivastava},
  year    = {2026},
  url     = {https://github.com/anishh15/Awesome-SLM-as-a-Judge}
}

---

🤝 Contributing

We welcome contributions from the community! If you'd like to add a paper, fix a link, or suggest improvements:

1. Open an Issue: Describe the paper or resource you'd like to add at github.com/anishh15/Awesome-SLM-as-a-Judge/issues.
2. Submit a Pull Request: Use the following format for new paper entries:

| ![Star](https://img.shields.io/github/stars/ORG/REPO.svg?style=social&label=Star) <br> [**Paper Title**](arxiv_link) | Venue | Year | [[pdf]](link) [[code]](link) |

Please make sure to include:

• ✅ Correct arXiv or venue link
• ✅ Code/model links if available
• ✅ Correct categorization within our taxonomy

---

📞 Contact & Support

• 📧 Email: anshladdha15@gmail.com, nitesh.pradhan@lnmiit.ac.in, gks@vt.edu
• 🐛 Issues: GitHub Issues

---

📄 License

MIT License -- see LICENSE for details.

---