SAT: Spatial Aptitude Training for Multimodal Language Models

Project Page

If you wish to use the latest versions of Huggingface and please check out the updated version of SAT here. It is the same dataset but compatible with the latest versions.

To use this dataset, first make sure you have Python3.10 and Huggingface datasets version 3.0.2 (pip install datasets==3.0.2).

from datasets import load_dataset
import io

split = "val"
dataset = load_dataset("array/SAT", batch_size=128)

example = dataset[split][10] # example 10th item

images = [Image.open(io.BytesIO(im_bytes)) for im_bytes in example['image_bytes']] # this is a list of images. Some questions are on one image, and some on 2 images

question = example['question']
answer_choices = example['answers'] # please randomize this if you present as choices to MLM, or perform circular eval.
correct_answer = example['correct_answer']

The available split choices are:

• train: (175K image QA pairs) Train split of SAT data that includes both static relationships and dyamic spatial QAs involving object and scene motion. For motion-based questions, there are two images.
• static: (127K image QA pairs) Train split of SAT data that includes only static QAs. Always has one image only.
• val: (4K image QA pairs) Synthetic validation split.
• test: (150 image QA pairs) Real-image dynamic test set. We highly recommend performing circular eval (test with all positions of the correct answer among answer choices) since this split is small. The paper results are with circular eval. Please find the lmms-eval code here

If you find this data useful, please consider citing:

@misc{ray2025satdynamicspatialaptitude,
      title={SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models}, 
      author={Arijit Ray and Jiafei Duan and Ellis Brown and Reuben Tan and Dina Bashkirova and Rose Hendrix and Kiana Ehsani and Aniruddha Kembhavi and Bryan A. Plummer and Ranjay Krishna and Kuo-Hao Zeng and Kate Saenko},
      year={2025},
      eprint={2412.07755},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.07755}, 
}