Dataset Card for TaskMeAnything-v1-imageqa-2024

TaskMeAnything-v1-imageqa-2024 benchmark dataset

🌐 Website | 📑 Paper | 🤗 Huggingface | 💻 Interface

If you like our project, please give us a star ⭐ on GitHub for latest update.

TaskMeAnything-v1-2024

TaskMeAnything-v1-imageqa-2024 is a benchmark for reflecting the current progress of MLMs by automatically finding tasks that SOTA MLMs struggle with using the TaskMeAnything Top-K queries. This benchmark includes 3,279 2d questions, 7,095 3d questions, and 1,896 real image questions that the TaskMeAnything algorithm automatically approximated as challenging for over 12 popular MLMs.

The dataset contains 19 splits, while each splits contains 600+ questions from a specific task generator in TaskMeAnything-v1. For each row of dataset, it includes: image, question, options, answer and its corresponding task plan.

Load TaskMeAnything-v1-2024 ImageQA Dataset

import datasets

dataset_name = 'weikaih/TaskMeAnything-v1-imageqa-2024'
dataset = datasets.load_dataset(dataset_name, split = TASK_GENERATOR_SPLIT)

where TASK_GENERATOR_SPLIT is one of the task generators, eg, 2024_2d_how_many.

Evaluation Results

Overall

Breakdown performance on each task types

Out-of-Scope Use

This dataset should not be used for training models.

Disclaimers

TaskMeAnything and its associated resources are provided for research and educational purposes only. The authors and contributors make no warranties regarding the accuracy or reliability of the data and software. Users are responsible for ensuring their use complies with applicable laws and regulations. The project is not liable for any damages or losses resulting from the use of these resources.

Contact

• Jieyu Zhang: jieyuz2@cs.washington.edu

Citation

BibTeX:

@article{zhang2024task,
  title={Task Me Anything},
  author={Zhang, Jieyu and Huang, Weikai and Ma, Zixian and Michel, Oscar and He, Dong and Gupta, Tanmay and Ma, Wei-Chiu and Farhadi, Ali and Kembhavi, Aniruddha and Krishna, Ranjay},
  journal={arXiv preprint arXiv:2406.11775},
  year={2024}
}