Dataset Card for LLaVA-CoT

The LLaVA-CoT-100k dataset is introduced in the paper LLaVA-CoT: Let Vision Language Models Reason Step-by-Step. This dataset is designed to enable Vision-Language Models (VLMs) to perform autonomous multistage reasoning, integrating samples from various visual question-answering sources with structured reasoning annotations. It aims to address the challenges VLMs face in systematic and structured reasoning for complex visual question-answering tasks.

Dataset Sources

• Repository: https://github.com/PKU-YuanGroup/LLaVA-CoT
• Paper: https://arxiv.org/abs/2411.10440

Sample Usage

You can easily load the dataset using the Hugging Face datasets library, and then follow the project's instructions for setting up images and using the data.

1. Load with Hugging Face datasets Library:

from datasets import load_dataset

# Load the LLaVA-CoT-100k dataset
dataset = load_dataset("Xkev/LLaVA-CoT-100k")

# Access the training split
train_split = dataset["train"]

# Print an example
print(train_split[0])

2. Prepare Images Locally:

The repository includes image.zip.part-{aa-ap} files. You need to merge these manually to get the full image archive:

cat image.zip.part-* > image.zip
unzip image.zip

3. Inference:

You can use the same code as Llama-3.2-11B-Vision-Instruct to load the model and perform inference. For detailed instructions on test-time stage-wise retracing search (SWIRES), refer to the inference/README.md file in the GitHub repository.

4. Finetuning:

To reproduce the paper's results, you can use the provided finetuning script with llama-recipes. Remember to modify the data_path and image_base_path in train/cot_dataset.py to your own local path to the training dataset.

cd train
pip install llama-recipes
torchrun --nnodes 1 --nproc_per_node 8 --master_port 29500 finetuning.py \
    --enable_fsdp --lr 1e-5 --num_epochs 3 --batch_size_training 4 \
    --model_name meta-llama/Llama-3.2-11B-Vision-Instruct \
    --dist_checkpoint_root_folder ./finetuned_model --dist_checkpoint_folder LLaVA-CoT \
    --use_fast_kernels --dataset "custom_dataset" --custom_dataset.test_split "test" \
    --custom_dataset.file "datasets/cot_dataset.py" --run_validation False \
    --batching_strategy padding

Dataset Structure

The train.jsonl file contains the question-answering data in the following format:

{
"id": ID,
"image": IMAGE_PATH,
"conversations": [{"from": "human", "value": QUESTION},{"from": "gpt", "value": ANSWER}]
}

Dataset Creation

We utilized images and questions from open-source datasets. The distribution is as follows:

Additionally, we used GPT-4o to generate structured answers. For details on the generation process, refer to dataset_generation/generate.py.

Bias, Risks, and Limitations

We have provided the sources of the images to the best of our ability. If you believe there is any infringement, please contact us immediately. We will remove the dataset and reference the provided links instead. The training images and questions are sourced from open datasets, and the answers are generated by GPT-4o. Despite our efforts to ensure diversity, some biases may still exist.

Citation

@misc{xu2024llavacot,
      title={LLaVA-CoT: Let Vision Language Models Reason Step-by-Step},
      author={Guowei Xu and Peng Jin and Hao Li and Yibing Song and Lichao Sun and Li Yuan},
      year={2024},
      eprint={2411.10440},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.10440},
}