📘 R-PRM Dataset (SFT + DPO)

This dataset is developed for training Reasoning-Driven Process Reward Models (R-PRM), proposed in our ACL 2025 paper. It consists of two stages:

• SFT (Supervised Fine-Tuning): collected from strong LLMs prompted with limited annotated examples, enabling reasoning-style evaluation.
• DPO (Direct Preference Optimization): constructed by sampling multiple reasoning trajectories and forming preference pairs without additional labels.

These datasets are used to train a generative reward model that performs step-by-step analysis and judgment of mathematical reasoning processes, improving both evaluation quality and guidance capabilities for policy models.

---

📁 Dataset Structure

R-PRM-dataset/
├── metadata.json
├── sft/
│   ├── train/         # 20 parquet files
│   └── validation/    # 20 parquet files
└── dpo/
    ├── train/         # 20 parquet files
    └── validation/    # 20 parquet files

Each split is sharded into 20 Parquet files for scalable processing. The metadata.json lists all file paths in structured form.

---

🔧 Usage Example

To load the SFT validation split from the remote Hugging Face Hub:

from datasets import load_dataset

dataset = load_dataset(
    "your-username/R-PRM-dataset",
    data_dir="sft/validation",
    split="train"
)

To load DPO training data:

dataset = load_dataset(
    "your-username/R-PRM-dataset",
    data_dir="dpo/train",
    split="train"
)

---

📄 Citation

If you use this dataset, please cite:

@misc{she2025rprmreasoningdrivenprocessreward,
      title={R-PRM: Reasoning-Driven Process Reward Modeling}, 
      author={Shuaijie She and Junxiao Liu and Yifeng Liu and Jiajun Chen and Xin Huang and Shujian Huang},
      year={2025},
      eprint={2503.21295},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.21295}, 
}