---

TriGap: Multimodal Continual Instruction Tuning Benchmark

TriGap is a challenging benchmark designed for Multimodal Large Language Models (MLLMs) in the context of Continual Instruction Tuning. It comprises instruction data derived from 10 diverse VQA datasets, covering domains such as medical imaging, scientific documents, autonomous driving, and abstract reasoning.

📚 Citation

If you use this benchmark in your research, please cite the following works:

@article{tang2026prism,
  title={Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning},
  author={Tang, Jun-Tao and Shi, Yu-Cheng and Xie, Zhen-Hao and Zhou, Da-Wei},
  journal={arXiv preprint arXiv:2605.26110},
  year={2026}
}

@inproceedings{xie2026same,
  title={SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning},
  author={Xie, Zhen-Hao and Tang, Jun-Tao and Shi, Yu-Cheng and Ye, Han-Jia and Zhan, De-Chuan and Zhou, Da-Wei},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2026}
}

📥 Dataset Download

The TriGap benchmark relies on image data from 10 underlying VQA datasets. After downloading the instruction data, you must download the corresponding images from their original sources.

Please refer to the Download Link column below to access the official repositories or project pages for each dataset.

📂 Directory Structure

To ensure compatibility with the TriGap benchmark loader, please organize your downloaded data into the following directory structure.

1. Image Data Structure

Place all downloaded images into an images folder, maintaining the specific sub-directory structure required by each dataset:

images
├── ArxivQA
├── CLEVR
│   ├── test
│   ├── train
│   └── val
├── ChartQA
├── ChemVQA
│   ├── test
│   └── train
├── DocVQA
│   └── spdocvqa_images
├── FloodNetVQA
│   ├── test_images
│   ├── train_images
│   └── valid_images
├── IconQA
│   ├── test
│   │   ├── choose_img
│   │   ├── choose_txt
│   │   └── fill_in_blank
│   ├── train
│   │   ├── choose_img
│   │   ├── choose_txt
│   │   └── fill_in_blank
│   └── val
│       ├── choose_img
│       ├── choose_txt
│       └── fill_in_blank
├── InfographicVQA
├── PMCVQA
└── Roadside
    ├── train_img
    └── val_img

2. Final Benchmark Structure

The final root directory for the TriGap benchmark should look like this:

TriGap/
├── images/          # Contains all the image sub-folders listed above
└── instructions/    # Contains the TriGap instruction JSON files