This dataset contains 1,000 high-quality synthetic verification chains-of-thought (CoTs) designed for training generative Process Reward Models (PRMs), as used in the paper "Process Reward Models That Think". The goal was to create a data-efficient alternative to traditional PRM training which often requires extensive human annotation or expensive rollouts.

Each instance consists of a math problem, a corresponding multi-step solution prefix (sourced from PRM800K [Lightman et al., 2023]), and a detailed verification CoT generated by the QwQ-32B-Preview. The verification CoT critiques each step of the solution prefix and provides a step-level correctness judgment (\boxed{correct} or \boxed{incorrect}).

To ensure high-quality synthetic CoTs, only chains where all step-level judgments matched the ground-truth human annotations from the PRM800K dataset were retained. They were also filtered based on correct formatting and length constraints to avoid issues like excessive overthinking observed in unfiltered generation. The figure below summarizes the synthetic cots collection. Refer to our paper for more details on data collection.

Curation Rationale

The dataset was created to enable efficient training of powerful generative PRMs. The core idea is that fine-tuning strong reasoning models on carefully curated, synthetic verification CoTs can yield verifiers that outperform models trained on much larger, traditionally labeled datasets. The process-based filtering (matching gold step labels) was shown to be crucial for generating high-quality training data compared to outcome-based filtering.

Code: https://github.com/mukhal/thinkprm

Paper: Process Reward Models that Think

Data Fields

The dataset contains the following fields:

• problem: (string) The mathematical problem statement (e.g., from MATH dataset via PRM800K).
• prefix: (string) The full step-by-step solution prefix being evaluated.
• cot: (string) The full synthetic verification chain-of-thought generated by QwQ-32B-Preview, including step-by-step critiques and judgments. See Fig. 13 in the paper for an example.
• prefix_steps: (list of strings) The solution prefix broken down into individual steps.
• gt_step_labels: (list of bools/ints) The ground-truth correctness labels (e.g., '+' for correct, '-' for incorrect) for each corresponding step in prefix_steps, sourced from PRM800K annotations.
• prefix_label: (bool) The overall ground-truth correctness label for the entire solution prefix: True if all steps are correct, False otherwise.

Source Data

• Problems & Solution Prefixes: Sourced from the PRM800K dataset, which is based on the MATH dataset.
• Verification CoTs: Generated synthetically using the QwQ-32B-Preview model prompted with instructions detailed in Fig. 14 of the paper.
• Filtering Labels: Ground-truth step-level correctness labels from PRM800K were used to filter the synthetic CoTs.

Citation Information

If you use this dataset, please cite the original paper:

@article{khalifa2025,
      title={Process Reward Models That Think}, 
      author={Muhammad Khalifa and Rishabh Agarwal and Lajanugen Logeswaran and Jaekyeom Kim and Hao Peng and Moontae Lee and Honglak Lee and Lu Wang},
      year={2025},
      journal={arXiv preprint arXiv:2504.16828},
      url={https://huggingface.co/papers/2504.16828}, 
}