Visual Perception Token Datasets

Dataset description

The training and evaluation datasets used in the Visual Perception Token.

We constructed the training dataset for Visual Perception Token based on the datasets from LLaVA-1.5 and visual-CoT. Our training data covers four types of tasks: Text/OCR-Related VQA, Spatial Reasoning, General VQA, and Fine-Grained VQA. The Text/OCR-Related VQA and Spatial Reasoning tasks are used to create training samples for Region Selection Token. The General VQA and Fine-Grained VQA tasks are used to construct training samples for DINO Feature Tokens.

The evaluation datasets includes the testing split of the training datasets and three zero-shot datasets, which are not included in the training.

Currently, we suggest cloning this repo or manually downloading the .json dataset files, which would meet the requirements for llama-factory training and evaluation.

If you wish to use the datasets library to process our dataset, a temporary workaround is to clone our dataset repository or manually download the .json files and then load each split locally using the datasets library.

# clone the dataset repo
git clone https://huggingface.co/datasets/rp-yu/VPT_Datasets

# load one dataset, e.g., MixVRT_CLIP_Full
from datasets import load_dataset
data = load_dataset("json", data_files="MixVRT_CLIP_Full.json")

License

The dataset is licensed under Apache 2.0