🧠 Sober Reasoning: Evaluation Logs

This repository hosts evaluation logs and outputs from our paper:

"A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility"

📄 Paper
📊 Leaderboard
💻 Evaluation Code

🗂️ Repository Structure

Evaluation logs are organized by the cluster used during inference to highlight hardware-induced variance in model performance (see Section 3.3 of the paper).

sober_reasoning/
├── cluster_A/
│   ├── <model_name>.csv
│   └── ...
├── cluster_B/
│   └── ...
└── cluster_C/
    └── ...

• <model_name>.csv: Contains accuracy (Pass@1), seeds, and temperature/top-p for every individual run.

🧪 Evaluation Coverage

We are uploading evaluation results for:

• Benchmarks: AIME'24, AIME'25, AMC'23, MATH500, Minerva, OlympiadBench and GPQA-Diamond

• Model categories:

  • RL-based models (e.g., OpenRS, DeepScaleR, Oat-Zero, etc.)
  • SFT-based models (e.g., OpenThinker, S1.1, Bespoke Stratos)
  • Baseline models (e.g., Qwen2.5 Base/Math)

📥 Coming Soon

• Full prompt-to-response logs per benchmark per model
• Code to reproduce the plots in the paper

🔄 Citation

@misc{hochlehnert2025soberreasoning,
      title={A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility}, 
      author={Andreas Hochlehnert and Hardik Bhatnagar and Vishaal Udandarao and Samuel Albanie and Ameya Prabhu and Matthias Bethge},
      year={2025},
      eprint={2504.07086},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2504.07086}, 
}