PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning

PhysReason is accepted by ACL-2025-main

📋 Overview

PhysReason is a comprehensive physics-based reasoning benchmark consisting of 1,200 physics problems spanning multiple domains, with a focus on both knowledge-based (25%) and reasoning-based (75%) questions. This benchmark addresses the critical gap in evaluating large language models' capabilities in physics-based reasoning, which requires applying physics theorems and constraints in complex problem-solving scenarios.

✨ Key Features

• 📊 Dataset Size: 1,200 carefully curated physics problems
• 🎯 Problem Types: Strategic mix of knowledge-based (25%) and reasoning-based (75%) questions
• 📚 Theorem Coverage: Comprehensive coverage of 147 physics theorems
• 🎨 Visual Content: 81% of problems include diagrams and visual elements
• 📈 Difficulty Levels: Four distinct levels - Knowledge, Easy, Medium, Hard
• 🔄 Step-by-step Solutions: Average of 8.1 solution steps per problem (15.6 for hard problems)
• 🌍 Multi-modal: Supports both text and image inputs

🔧 Data Collection

Our rigorous data collection process ensures high-quality, challenging problems:

• 📖 Sources: Global college entrance exams and international physics competitions
• ⚙️ Process: Standardized using MinerU framework for consistent formatting
• ✅ Quality Control: Two-phase translation process with expert verification
• 🔍 Filtering: Systematically excluded easily searchable problems to prevent data leakage
• 📊 Classification: Difficulty levels based on solving time and theorem complexity analysis

📊 Benchmark Comparison

🔍 Evaluation Framework

We introduce the Physics Solution Auto Scoring (PSAS) framework with two complementary evaluation approaches:

PSAS-A (Answer Level Evaluation)

• Sub-question Assessment: Evaluates answers for each sub-question independently
• LLM-based Extraction: Uses advanced language models for answer extraction
• Semantic Verification: Ensures semantic consistency between extracted and ground truth answers
• Weighted Scoring: Considers solution step lengths as weights for different sub-questions

PSAS-S (Step Level Evaluation)

Provides detailed step-by-step assessment through four phases:

1. Data Extraction: Parses model responses and reference solutions
2. Scoring: Evaluates correctness of each reasoning step
3. First Error Detection: Identifies where models first deviate from correct reasoning
4. Error Analysis: Classifies error types into four key bottlenecks:
   • Physics Theorem Application
   • Physics Process Understanding
   • Calculation
   • Physics Condition Analysis

🚀 Usage

Core Evaluation Files

• answer_evaluation_with_ds_ch_prompt.py: Answer-level evaluation using Chinese prompts
• answer_evaluation_with_ds_en_prompt.py: Answer-level evaluation using English prompts
• format_result_ds.py: Optimizes unstable outputs into stable, consistent formats
• step_evaluation_with_ds_ch_prompt.py: Step-level evaluation using Chinese prompts
• step_evaluation_with_ds_en_prompt.py: Step-level evaluation using English prompts

📈 Experimental Results

Non-O-like Models Performance

O-like Models Performance

PhysReason-mini Results

🔑 Key Findings

• Performance Gap: Even top-performing models achieve less than 60% on answer-level evaluation
• Difficulty Scaling: Performance drops significantly from knowledge questions (75.11%) to hard problems (31.95%)
• O-like Model Advantage: Models with enhanced reasoning capabilities show superior performance
• Multi-modal Benefits: Visual content significantly enhances model understanding and performance
• Four Critical Bottlenecks identified through step-level evaluation:
  1. Physics Theorem Application
  2. Physics Process Understanding
  3. Calculation Accuracy
  4. Physics Condition Analysis

📝 Citation

If you find PhysReason useful in your research, please cite our paper:

@article{zhang2025physreason,
  title={Physreason: A comprehensive benchmark towards physics-based reasoning},
  author={Zhang, Xinyu and Dong, Yuxuan and Wu, Yanrui and Huang, Jiaxing and Jia, Chengyou and Fernando, Basura and Shou, Mike Zheng and Zhang, Lingling and Liu, Jun},
  journal={arXiv preprint arXiv:2502.12054},
  year={2025}
}

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📧 Contact

We welcome contributions to PhysReason! Please contact us for more details.

---

🔗 Quick Links:

• 📄 Paper
• 🤗 Dataset
• 🌐 Project Page
• 💻 GitHub Repository