PRISM Public SFT Data

Overview

PRISM Public SFT Data is the public supervised fine-tuning data collection used in the PRISM project.

PRISM studies the distributional drift problem in the standard SFT → RLVR post-training pipeline for large multimodal models. Before the distribution alignment and RLVR stages, we first use large-scale public multimodal demonstrations to obtain a broad SFT initialization.

This dataset serves as the public SFT data source for the first stage of PRISM:

SFT → Distribution Alignment / Pre-alignment → RLVR

Dataset Description

This dataset contains public multimodal SFT examples collected and formatted for training large vision-language models.

The examples are designed to cover diverse multimodal instruction-following and reasoning scenarios, typically including:

• an input image;
• a question or instruction;
• a supervised response;
• multimodal reasoning or visual-question-answering style supervision.

This dataset is used to provide a broad and general-purpose SFT initialization before PRISM applies distribution alignment and downstream RLVR.

Role in PRISM

In PRISM, this public SFT data is used in the first training stage to adapt the base multimodal model to instruction-following and multimodal reasoning tasks.

After this SFT stage, PRISM further applies a distribution alignment / pre-alignment stage with black-box on-policy distillation, followed by RLVR. The overall training pipeline is:

Public SFT Data → SFT Model → PRISM Pre-alignment → RLVR Model

This dataset should be viewed as the broad public supervision source for SFT, while the Gemini-distilled data provides higher-fidelity supervision for the later PRISM pre-alignment stage.

Intended Use

This dataset is intended for research on:

• supervised fine-tuning of vision-language models;
• multimodal instruction tuning;
• multimodal reasoning;
• visual question answering;
• post-training for large multimodal models;
• SFT initialization before RLVR.

Dataset Statistics

Usage

For data processing, training, inference, and evaluation, please refer to our GitHub repository:

https://github.com/XIAO4579/PRISM

We recommend using the provided scripts and environment to reproduce the PRISM training pipeline.

Citation

If you find PRISM or this dataset useful, please cite our paper:

@misc{wang2026sfttorlprealignmentblackboxonpolicy,
      title={Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL}, 
      author={Sudong Wang and Weiquan Huang and Xiaomin Yu and Zuhao Yang and Hehai Lin and Keming Wu and Chaojun Xiao and Chen Chen and Wenxuan Wang and Beier Zhu and Yunjian Zhang and Chengwei Qin},
      year={2026},
      eprint={2604.28123},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.28123}, 
}

Acknowledgements

We gratefully acknowledge the open-source datasets and projects that made this work possible.

We also thank LLaMA-Factory, verl, and lmms-eval for their excellent open-source infrastructure.