VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search
44
110 commits
2 linked in READMEs
updated Feb 1, 2026
VisualWebInstruct is a large-scale, diverse multimodal instruction dataset designed to enhance vision-language models' reasoning capabilities. The dataset contains approximately 900K question-answer (QA) pairs, with 40% consisting of visual QA pairs associated with 163,743 unique images, while the remaining 60% are text-only QA pairs.
Please also checkout our more recent verified version at Huggingface.
This dataset is referenced in the survey paper A Survey of Deep Learning for Geometry Problem Solving.
VisualWebInstruct addresses the critical challenge of scarce high-quality, diverse training data for multimodal reasoning tasks. Unlike existing datasets that focus narrowly on specific domains or rely on synthetic images, VisualWebInstruct offers:
The repository is organized into three main subsets:
images.zip fileThe dataset was created through an innovative approach:
Models fine-tuned on VisualWebInstruct demonstrate substantial improvements:
The following table depicts the sources for our images:
| Image Source | Count |
|---|---|
| visualwebinstruct | 118,576 |
| coco | 32,606 |
| chartqa | 17,008 |
| ai2d | 11,397 |
| geoqa+ | 11,364 |
| sqa | 5,650 |
| vg | 4,767 |
| docvqa | 3,978 |
| gqa | 3,529 |
| ocr_vqa | 4,448 |
| textvqa | 1,200 |
| others | 2,635 |
To use this dataset for training or fine-tuning:
from datasets import load_dataset
# Load the complete dataset
dataset = load_dataset("TIGER-AI-Lab/VisualWebInstruct")
# Load specific subsets
conversation_data = load_dataset("TIGER-AI-Lab/VisualWebInstruct", "conversation")
qa_data = load_dataset("TIGER-AI-Lab/VisualWebInstruct", "visualwebinstruct")
# Access images
# The imgs.zip file needs to be downloaded and extracted separately
If you use VisualWebInstruct in your research, please cite our paper:
@article{visualwebinstruct,
title = {VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search},
author = {Jia, Yiming and Li, Jiachen and Yue, Xiang and Li, Bo and Nie, Ping and Zou, Kai and Chen, Wenhu},
journal = {arXiv preprint arXiv:2503.10582},
year = {2025}
}
This research was supported by NetMind.Ai for providing cloud compute and Google DeepMind for generous support for Gemini credits.
VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search
44
110 commits
2 linked in READMEs
updated Feb 1, 2026
VisualWebInstruct is a large-scale, diverse multimodal instruction dataset designed to enhance vision-language models' reasoning capabilities. The dataset contains approximately 900K question-answer (QA) pairs, with 40% consisting of visual QA pairs associated with 163,743 unique images, while the remaining 60% are text-only QA pairs.
Please also checkout our more recent verified version at Huggingface.
This dataset is referenced in the survey paper A Survey of Deep Learning for Geometry Problem Solving.
VisualWebInstruct addresses the critical challenge of scarce high-quality, diverse training data for multimodal reasoning tasks. Unlike existing datasets that focus narrowly on specific domains or rely on synthetic images, VisualWebInstruct offers:
The repository is organized into three main subsets:
images.zip fileThe dataset was created through an innovative approach:
Models fine-tuned on VisualWebInstruct demonstrate substantial improvements:
The following table depicts the sources for our images:
| Image Source | Count |
|---|---|
| visualwebinstruct | 118,576 |
| coco | 32,606 |
| chartqa | 17,008 |
| ai2d | 11,397 |
| geoqa+ | 11,364 |
| sqa | 5,650 |
| vg | 4,767 |
| docvqa | 3,978 |
| gqa | 3,529 |
| ocr_vqa | 4,448 |
| textvqa | 1,200 |
| others | 2,635 |
To use this dataset for training or fine-tuning:
from datasets import load_dataset
# Load the complete dataset
dataset = load_dataset("TIGER-AI-Lab/VisualWebInstruct")
# Load specific subsets
conversation_data = load_dataset("TIGER-AI-Lab/VisualWebInstruct", "conversation")
qa_data = load_dataset("TIGER-AI-Lab/VisualWebInstruct", "visualwebinstruct")
# Access images
# The imgs.zip file needs to be downloaded and extracted separately
If you use VisualWebInstruct in your research, please cite our paper:
@article{visualwebinstruct,
title = {VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search},
author = {Jia, Yiming and Li, Jiachen and Yue, Xiang and Li, Bo and Nie, Ping and Zou, Kai and Chen, Wenhu},
journal = {arXiv preprint arXiv:2503.10582},
year = {2025}
}
This research was supported by NetMind.Ai for providing cloud compute and Google DeepMind for generous support for Gemini credits.