🌮 TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action

🌐 Website | 📑 Arxiv | 💻 Code| 🤗 Datasets

If you like our project or are interested in its updates, please star us :) Thank you! ⭐

Summary

TLDR: CoTA is a large-scale dataset of synthetic Chains-of-Thought-and-Action (CoTA) generated by programs.

Load data

from datasets import load_dataset
dataset = load_dataset("Salesforce/program-cota-llava", split="program_cota_mc_970k")

Dataset Card

Dataset Details

This dataset contains synthetic chains of thoughts and actions.

Uses

The intended use of this dataset is to finetune multi-modal language models to produce chains of thoughts and actions to answer difficult and complex visual questions.

Direct Use

You can directly use this dataset to train LLaVA-OneVision-based models with our codebase. To train Mantis models, please use program-cota-mantis in the collection. To train other multi-modal language models, you might need to adapt the conversation format to work for your particular models.

Out-of-Scope Use

This dataset should not be used for testing models.

Source Data

The source data comes from Cauldron and Mantis-Instruct. They are collected from various existing datasets, including COCO, AOKVQA, ScienceQA, Visual Genome, etc.

Data Collection and Processing

Bias, Risks, and Limitations

Our dataset has the following limitations:

• The chains of thoughts and actions are generated by gpt-4o-2024-08-06 and thus inherit its biases;
• The actions are somewhat limited as they cover mostly vision-centric tools such as DepthEstimation and some generic tools such as QueryKnowledgeBase.
• Please refer to the paper for additional limitations.

License

The CoTA datasets are licensed under the noncommerical license CC-BY-NC 4.0. Users need to make their own assessment regarding any obligations or responsibilities under the corresponding licenses or terms and conditions pertaining to the original datasets and data. This release is for research purposes only in support of an academic paper.

Citation

@misc{ma2024tacolearningmultimodalaction,
      title={TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action}, 
      author={Zixian Ma and Jianguo Zhang and Zhiwei Liu and Jieyu Zhang and Juntao Tan and Manli Shu and Juan Carlos Niebles and Shelby Heinecke and Huan Wang and Caiming Xiong and Ranjay Krishna and Silvio Savarese},
      year={2024},
      eprint={2412.05479},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.05479}, 
}