Overview

🌐 Website • 🤗 Hugging Face • ⏬ Data • 📃 Paper • 🖥️ Github

💥 DeltaBench

DeltaBench is the first dataset to analyze the qualities of the long CoTs generated by o1-like models and evaluate the critique abilities to Detect Error in Long CoT ReAsoning of existing critic models and PRMs. Specifically, DeltaBench comprises 1,236 samples across diverse domains, including Math, Programming, PCB (physics, chemistry and biology), and General Reasoning. Each sample encompasses a problem, its corresponding long CoT solution, and comprehensive human annotations

Please visit our website or check our paper for more details.

🆕 News

• [Soon] We plan to release more of our labeled datasets, which will be available for training and research purposes. Stay tuned 🔥🔥🔥

• [2025.03.05] We have released the DeltaBench dataset 🤗huggingface 🚀🚀🚀.

💫 Introduction

• We introduce DeltaBench, the first dataset to analyze the qualities of the long CoTs generated by o1-like models and evaluate the critique abilities to Detect Error in Long CoT ReAsoning of existing critic models and PRMs. Specifically, in DeltaBench, we first collect a diverse collection of long CoTs generated by various o1-like models (i.e., QwQ, DeepSeek-R1, and Gemini-2.0 Flash Thinking) across different reasoning tasks such as Math, Programming, PCB (physics, chemistry and biology), and General Reasoning. Then, we divide each long COT into different sections, where each section denotes an independent subtask, as shown in Figure 1.

Figure 1: An example of dividing sections.

​ After that, each section includes the following tags:

• 1️⃣Strategy Shift: whether this section introduces a new method or strategy attempt. If a new strategy is introduced, the specific step is annotated.
• 2️⃣Reasoning Usefulness: whether the reasoning in this section is useful.
• 3️⃣Reasoning Correctness: whether this section contains any errors. If an error is present, additional error-related fields are annotated, including the first step number at which the error occurs, explanation and correction.
• 4️⃣Reflection Efficiency: whether this section contains reflection and whether the reflection is correct. If reflection is present, the step at which the reflection begins is annotated.

📊 Leaderboard

⚖️ Evals

The startup command is as follows:

• Step 1: set your openai key in scripts/chinese_simpleqa_easy.py:

  os.environ["OPENAI_API_KEY"] = "replace your key here"
  os.environ["OPENAI_BASE_URL"] = "replace your key here"
  

• Step 2: run the eval script:

  python evaluation.py <model_name>
  

📜 Citation

Please cite our paper if you use our dataset.

@misc{he2025largelanguagemodelsdetect,
      title={Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?}, 
      author={Yancheng He and Shilong Li and Jiaheng Liu and Weixun Wang and Xingyuan Bu and Ge Zhang and Zhongyuan Peng and Zhaoxiang Zhang and Zhicheng Zheng and Wenbo Su and Bo Zheng},
      year={2025},
      eprint={2502.19361},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.19361}, 
}