IndicVisionBench

This repository contains the dataset for IndicVisionBench, introduced in

“IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs”
📄 arXiv:2511.04727
🏛️ Accepted at ICLR 2026
🔗 OpenReview: https://openreview.net/forum?id=LmJoLn04iL

IndicVisionBench is a culturally grounded, multilingual vision-language benchmark designed to evaluate Vision–Language Models (VLMs) on visual understanding tasks in the Indian context. The benchmark focuses on:

• Multilingual Visual Question Answering (VQA)
• Culturally-aware reasoning
• Adversarial robustness
• Parallel cross-lingual consistency
• Optical Character Recognition (OCR) in Indic scripts
• Multimodal Machine Translation (MMT)

Unlike generic VQA datasets, IndicVisionBench emphasizes Indian cultural context, regional diversity, and Indic language coverage, enabling systematic evaluation of multilingual and culturally-aware VLMs.

---

Languages Covered

• English
• Hindi
• Tamil
• Telugu
• Malayalam
• Marathi
• Gujarati
• Punjabi
• Odia
• Kannada
• Bengali

---

Benchmark Overview

IndicVisionBench consists of five main configurations:

• Total images across all configs: 4993
• Total questions across VQA En, Indic and Parallel: (4117 + 1007 + 1166)*6 = 37,740

---

Subset Descriptions

1️⃣ Multimodal Machine Translation (mmt)

Image-grounded translation benchmark with aligned captions across multiple Indic languages.

Features:

• image
• topic
• State/UT
• Parallel captions in 11 languages
• source_url

This subset evaluates:

• Cultural terminology consistency
• Visual grounding in translation

2️⃣ Optical Character Recognition (ocr)

OCR dataset consisting of scanned pages in Indic scripts from Wikisource.

Features:

• image
• text
• language
• page_url

This subset evaluates OCR capabitilies on Indic scripts/languages.

3️⃣ English VQA (vqa_en)

Culturally grounded VQA in English.

Each example includes:

• 2 short-answer questions
• 1 multiple-choice question (4 options)
• 1 true/false question
• 1 long-form reasoning question
• 1 adversarial question
• Metadata: topic, language, State/UT, 'source_url'

This subset evaluates:

• Object & scene understanding
• Cultural knowledge
• Fine-grained attribute recognition
• Robustness to false assumptions in the adversarial questions

4️⃣ Indic VQA (vqa_indic)

Same VQA format as in vqa_en, but in Indic languages.

This subset evaluates:

• Multilingual reasoning
• Cultural alignment in local languages

5️⃣ Parallel VQA (vqa_parallel)

Same VQA format as in vqa_en. Parallel multilingual QA pairs for the same image.

This subset enables the study of

• cross-lingual performance of VLMs across 11 languages (English and 10 Indic languages)
• region-specific strengths or biases

Usage

All configurations can be loaded using datasets:

from datasets import load_dataset

# Example: load English VQA split
ds = load_dataset("krutrim-ai-labs/IndicVisionBench", "vqa_en")["test"]

print(ds[0])

The following five configurations/splits are present in the dataset:

• mmt
• ocr
• vqa_en
• vqa_indic
• vqa_parallel

Images are stored directly within the dataset and loaded automatically by 🤗 Datasets.

Evaluation Dimensions

IndicVisionBench is designed to measure:

• Scene & contextual understanding
• Attribute detection
• Cultural understanding
• Bias & adversarial robustness
• Cross-lingual consistency
• OCR performance
• Image-grounded translation capability

Code & Evaluation

The official inference and evaluation codebase for IndicVisionBench is available on GitHub.

GitHub Repository: https://github.com/ola-krutrim/IndicVisionBench

The repository provides the complete pipeline for running inference and reproducing benchmark results across all evaluation tracks.

The codebase includes:

• End-to-end inference pipelines for Vision-Language Models (VLMs) and OCR systems
• Modular wrappers enabling easy integration of API-based models and open-source models
• Evaluation pipelines for all benchmark tasks:
  • OCR evaluation
  • Visual Question Answering (VQA)
    • Structured questions (MCQ, True/False)
    • Open-ended questions (short answer, long answer, adversarial)
  • Multimodal Machine Translation (MMT)
• LLM-as-a-judge evaluation for open-ended VQA responses
• Data generation scripts for constructing a similar multimodal benchmark.

Citation

If you use this dataset, please cite:

@inproceedings{faraz2026indicvisionbench,
  title={IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs},
  author={Ali Faraz and Akash and Shaharukh Khan and Raja Kolla and Akshat Patidar and Suranjan Goswami and Abhinav Ravi and Chandra Khatri and Shubham Agarwal},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2026},
  url={https://openreview.net/forum?id=LmJoLn04iL}
}