Dataset Card for VRC-Bench

Dataset Sources

• Repository: [https://github.com/mbzuai-oryx/LlamaV-o1]

• Paper*

Dataset Structure

Each data sample contains following field:

{
    "image": PIL.Image
    "question": "What is the difference of largest and smallest bar?",
    "idx": "MathVista_74",
    "final_answer": "47.6",
    "steps": [
        "Step 1: Identify the largest bar in the chart.  \nAction 1: The largest bar is for Iceland at 100%.",
        "\nStep 2: Identify the smallest bar in the chart.  \nAction 2: The smallest bar is for Cambodia at 52.4%.",
        "\nStep 3: Calculate the difference between the largest and smallest bars.  \nAction 3: Subtract the smallest percentage from the largest percentage: 100% - 52.4% = 47.6%."
    ]
}

Dataset Overview

The figure presents our benchmark structure and the comparative performance of LMMs on VRC-Bench. The dataset spans diverse domains, including mathematical & logical reasoning, scientific reasoning, visual perception, and specialized areas such as medical imaging, cultural understanding, and document OCR. It also includes tasks like chart & diagram comprehension to test real-world applications. The bar chart compares various state-of-the-art models, showcasing final answer accuracy and step-by-step reasoning performance. Our [LlamaV-o1](https://huggingface.co/omkarthawakar/LlamaV-o1) model surpasses GPT-4o-mini, Gemini-1.5-Flash, and Llava-CoT in complex multimodal reasoning tasks, achieving superior accuracy and logical coherence.

Dataset Examples

---

Results

Table: Comparison of models based on Final Answer accuracy and Reasoning Steps performance on the proposed VRC-Bench. The best results in each case (closed-source and open-source) are in bold. Our LlamaV-o1 achieves superior performance compared to its open-source counterpart (Llava-CoT) while also being competitive against the closed-source models.

---

Breakdown for VCR-Bench Categories

---

Load Dataset

from datasets import load dataset
ds = load_dataset("omkarthawakar/VRC-Bench", split="test")

Citation

If you find this paper useful, please consider staring 🌟 our Github repo and citing 📑 our paper:

@misc{thawakar2025llamavo1,
      title={LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs}, 
      author={Omkar Thawakar and Dinura Dissanayake and Ketan More and Ritesh Thawkar and Ahmed Heakl and Noor Ahsan and Yuhao Li and Mohammed Zumri and Jean Lahoud and Rao Muhammad Anwer and Hisham Cholakkal and Ivan Laptev and Mubarak Shah and Fahad Shahbaz Khan and Salman Khan},
      year={2025},
      eprint={2501.06186},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2501.06186}, 
}