VGR-SFT: A Dataset for Visual Grounded Reasoning

• ArXiv Paper
• Dataset on Hugging Face

Dataset Overview

VGR-SFT (Visual Grounded Reasoning—Supervised Fine-Tuning) is a large-scale multimodal reasoning dataset released with the paper “VGR: Visual Grounded Reasoning”. It is an early attempt to explicitly model visual-region attention in multimodal reasoning, combining vision grounding with language-based deduction. It enables models to autonomously attend to arbitrary visual regions during the reasoning process.

Key Features

• Joint Visual-Language Reasoning: Each sample contains an image, a question, a reasoning chain, and an answer, along with region annotations that support the reasoning.
• Autonomous Region Attention: Grounding regions are generated by models, reducing manual-annotation bias.
• Diverse Domain Coverage: Covers multiple task types, such as science question answering, chart understanding, and document visual question answering.
• Efficient Feature Utilization: Reduces visual-token consumption by ~70% compared to baselines via a selective feature replay mechanism.

Dataset Structure

Data Composition

To stay aligned with the baseline, all data are collected from LLaVA-NeXT. Due to copyright restrictions, we do not distribute the source images directly. Please download the required images from the official LLaVA-NeXT dataset. Images in the llava_next_raw_format directory follow the same relative paths as the image field in VGR-SFT.

Data Construction Pipeline

1. Cold-Start Data Generation: Generate initial reasoning data with visual-region annotations using Qwen2.5-VL-72B.
2. Reject Sampling Refinement:
   • Format Verification: Ensure answers are parseable and coordinates are correctly formatted.
   • Correctness Verification: Validate reasoning accuracy via ANLS and commercial model APIs.
   • Visual Grounding Verification: Crop regions and verify content alignment with annotations.
3. Data Scaling: Train an annotation model using InternVL3-14B, integrate Open-R1 text-reasoning data to improve generalization, and rewrite training data with a commercial model.

Model Performance with VGR

Data Example

Citation

If you use this dataset, please cite the following paper:

@article{wang2025vgr,
  title={VGR: Visual Grounded Reasoning}, 
  author={Jiacong Wang and Zijian Kang and Haochen Wang and Haiyong Jiang and Jiawen Li and Bohong Wu and Ya Wang and Jiao Ran and Xiao Liang and Chao Feng and Jun Xiao},
  journal={arXiv preprint arXiv:2506.11991},
  year={2025}
}

License

This dataset is released under the Creative Commons Zero v1.0 Universal (CC-0) license, subject to any intellectual property rights in the dataset owned by Bytedance. The data are adapted from the LLaVA-NeXT project; use of that data must comply with its respective licenses. Please see the disclaimer for more details.