Dataset Card for VL-PRM300K

Table of Contents

• Table of Contents
• Dataset Description
  • Dataset Summary
• Dataset Structure
  • Data Fields
• Additional Information
  • Citation Information

Dataset Description

• Repository: https://github.com/theogbrand/vlprm
• Paper: Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned

Dataset Summary

VL-PRM300K is a dataset of 300,000 samples of step-level solutions to a set of diverse and difficult visual reasoning tasks for training Vision Language Process Reward Models (VL-PRMs) with distilled reasoning traces from GPT-4.1 and judge solutions from o4-mini. Refer to the VL-PRMs paper for more details.

Usage

# pip install -q datasets
from datasets import load_dataset
ds = load_dataset("ob11/VL-PRM300K")["train"]

Dataset Structure

Data Fields

The data fields are:

• id: Unique ID assigned to training sample
• conversations: Reasoning trace formatted for running training, based on first_incorrect_step
• first_incorrect_step: List containing two elements. The first element is either "Visual Elements"/"Reasoning", indicating the type of step-level error detected in the trace by o4-mini judge model; The second element contains the index of the step, corresponding to the list item in the column steps_with_score, zero-indexed. e.g. ["Visual Elements", 4] means the fourth step in the steps_with_score list contains the first error. Important: ["Reasoning", 0] means the FIRST reasoning step contains the error. To find the corresponding step from "steps_with_score" we need to first get the total number of steps in the Visual Elements section, then add the Reasoning index. We will support a "direct" reasoning index step in the future.
• steps_with_score: List of step-by-step traces generated by GPT-4.1, with all prompt format tags removed.
• rollout_ground_truth_answer: The ground truth solution to the question
• question_type: High level reasoning category the question falls into
• source: The dataset the question came from
• question: The question
• consensus_filtering_algo_label: Label output from our Consensus Filtering Algorithm
• verifier_identified_first_incorrect_step_solution: The reasoning trace generated by o4-mini to identify the incorrect step (if present)
• image: The absolute image path of the original image from source when you untar the files in the image directory
• image_qwen_smart_resize: The absolute image path of the original image which was postprocessed using Qwen-2.5-VL image resizing algorithm

Citation Information

@misc{ong2025vlprms,
      title={Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned}, 
      author={Brandon Ong, Tej Deep Pala, Vernon Toh, William Chandra Tjhi, and Soujanya Poria},
      year={2025},
      eprint={2509.23250},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/pdf/2509.23250}, 
}