SliME SMR data card

Dataset details

Dataset type:

Generation of Source Data and Instruction Data. The creation of SMR involves a meticulous amalgamation of publicly available datasets, comprising Arxiv-QA, ScienceQA, MATH-Vision, TextBookQA, GeoQA3, Geometry3K, TabMWP, DVQA, AI2D, and ChartVQA.

The disparities between SMR and conventional instruction tuning datasets manifest in two key aspects:

1. Challenging Reasoning Tasks: Many of the tasks in Physical/Social science and mathematics demand advanced reasoning abilities.

2. Demand for Image Detail Understanding: All tasks necessitate a profound understanding of visual details because many images contain rich annotation information or questions requiring comprehensive visual analysis.

To ensure the accuracy of our data, we carefully filter it after collection. This involves identifying and fixing issues like blurry images or jumbled text, unrelated image-text pairs, and incorrect reasoning paths that can't lead to correct answers or might lead to wrong conclusions. For the latter, we use GPT-4V to create new, accurate reasoning paths.

SMR data structure

data
├── arxivqa
│   └── images
├── DVQA
│   └── images
├── Geometry3K
│   └── 0-2400 dirs
├── ChartQA
│   └── train_images
└── GeoQA3
│    ├── image
│    └── json
├── mathvision
├── scienceqa
├── tabmwp
└── GeoQA3
│    ├── train
│    └── test
│    └── val
└── ai2d
│    ├── abc_images
│    └── images
└── geoqa+
│   └── images

1. Arxiv QA Download images using this download url

python playground/data/process_arxivqa.py

2. DVQA

Download images using this url.

3. ChartQA

Clone this repo

extract all the training images in ChartQA_Dataset/train/png into ChartQA

4. Geometry3K

Download images using this url.

The image path in our json file will be os.path.join(f'Geometry3K/i', 'img_diagram.png')

5. GeoQA3

Download images using this url

extract all the training images in GeoQA3/image

6. MathVision

Download images using this url

Our data will not include the images from test-mini split automatically

7. ScienceQA

wget https://scienceqa.s3.us-west-1.amazonaws.com/images/train.zip
wget https://scienceqa.s3.us-west-1.amazonaws.com/images/val.zip
wget https://scienceqa.s3.us-west-1.amazonaws.com/images/test.zip

unzip -q train.zip
unzip -q val.zip
unzip -q test.zip

rm train.zip
rm val.zip
rm test.zip

8. Tabmwp

Download images using this url

9. TextbookQA

Download images using this url

10. AI2D:

Download images using this url

11. GeoQA+

Download images using this url

Paper or resources for more information:

Paper: https://huggingface.co/papers/2406.08487

Arxiv: https://arxiv.org/abs/2406.08487

Code: https://github.com/yfzhang114/SliME

License: Attribution-NonCommercial 4.0 International It should abide by the policy of OpenAI: https://openai.com/policies/terms-of-use

Intended use

Primary intended uses: The primary use of SMR is research on large multimodal models and chatbots.

Primary intended users: The primary intended users of this dataset are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.