MirageTVQA: Multilingual Visual Table Question Answering Benchmark

📋 Table of Contents

• Overview
• Dataset Statistics
• Dataset Structure
• Usage
• Languages
• Citation

Overview

MirageTVQA is the first large-scale multilingual visual table question-answering benchmark designed to evaluate Vision-Language Models (VLMs) on realistic table understanding tasks. Unlike existing benchmarks that focus solely on clean English tables, MirageTVQA challenges models with:

• 45 languages spanning 10+ language families
• 108,772 question-answer pairs across diverse reasoning categories
• 40,730 unique table images with both clean and noisy variants
• Real-world noise simulating scanned documents and camera captures
• 10 reasoning types from numerical aggregation to multi-hop reasoning

Our evaluation reveals two critical failure modes in current VLMs:

1. 35%+ performance drop on noisy images (even for best models)
2. Severe English-first bias with poor cross-lingual transfer

📊 Dataset Statistics

Overall Statistics

• Total Questions: 108,772
• Unique Tables: 10,183
• Total Images: 40,730
  • Clean images: 10,183
  • Noisy variants: 30,547 (3 variants per table)
• Languages: 45
• Language Families: 10+ (Indo-European, Sino-Tibetan, Afro-Asiatic, Austronesian, Japonic, Koreanic, Kra-Dai, Turkic, Dravidian)

Per-Language Statistics

Reasoning Categories Distribution

• Comparative Reasoning
• Numerical Aggregation
• Multi-Hop Reasoning
• Temporal Reasoning
• Conditional Reasoning
• Proportional/Ratio Analysis
• Hypothetical Reasoning
• Correlation Inference
• Structural/Metadata Reasoning
• Outlier Detection

📁 Dataset Structure

Data Files

MirageTVQA/
├── miragetvqa_eval.jsonl          # All QA pairs with metadata
└── images.zip                      # All table images
    └── images/
        └── {table_id}/
            ├── clean/
            │   └── {lang_code}_clean.jpg
            └── noisy/
                ├── {lang_code}_noise1.jpg
                ├── {lang_code}_noise2.jpg
                └── {lang_code}_noise3.jpg

JSONL Format

Each line in miragetvqa_eval.jsonl contains:

{
  "question_id": "finqa_cdb26d6873_006",
  "table_id": "finqa_cdb26d6873",
  "language": "te",
  "language_name": "Telugu",
  "language_family": "Dravidian",
  "question": "మొదటి కాలానికి సంబంధించిన 'ఇతర సమగ్ర ఆదాయం (నష్టం)' (వరుస 5) లో, '$606 మిలియన్ల' మొత్తం నష్టంలో 'అందుబాటులో ఉన్న పెట్టుబడులపై వాస్తవికం కాని హోల్డింగ్ లాభాలు (నష్టాలు)'కి ఎంత శాతం నష్టం ఆపాదించబడింది?",
  "answer": [["117.16%"]],
  "question_type": "value",
  "reasoning_category": "Proportional/Ratio Analysis",
  "evidence_cells": ["B5", "H5"]
}

Field Descriptions

Usage

Loading the Dataset

from datasets import load_dataset
import zipfile
import json

# Load the JSONL data
dataset = load_dataset("your-username/MirageTVQA", split="test")

# Or load directly from JSONL
data = []
with open("miragetvqa_eval.jsonl", "r", encoding="utf-8") as f:
    for line in f:
        data.append(json.loads(line))

# Extract images
with zipfile.ZipFile("images.zip", "r") as zip_ref:
    zip_ref.extractall(".")

Evaluating a Model

from PIL import Image
import os

def evaluate_model(model, language="en", use_noisy=False):
    results = []
    
    for item in data:
        if item["language"] != language:
            continue
        
        # Construct image path
        table_id = item["table_id"]
        lang_code = item["language"]
        
        if use_noisy:
            img_path = f"images/{table_id}/noisy/{lang_code}_noise1.jpg"
        else:
            img_path = f"images/{table_id}/clean/{lang_code}_clean.jpg"
        
        # Load image
        image = Image.open(img_path)
        
        # Get model prediction
        prediction = model.predict(image, item["question"])
        
        # Calculate exact match
        is_correct = prediction in item["answer"][0]
        results.append(is_correct)
    
    accuracy = sum(results) / len(results) * 100
    return accuracy

# Evaluate on clean English images
clean_acc = evaluate_model(your_model, language="en", use_noisy=False)
print(f"Clean accuracy: {clean_acc:.2f}%")

# Evaluate on noisy English images
noisy_acc = evaluate_model(your_model, language="en", use_noisy=True)
print(f"Noisy accuracy: {noisy_acc:.2f}%")

Language-Specific Evaluation

# Evaluate across all languages
language_scores = {}
for lang_info in dataset.unique("language"):
    lang_code = lang_info
    score = evaluate_model(your_model, language=lang_code)
    language_scores[lang_code] = score

# Print results
for lang, score in sorted(language_scores.items(), key=lambda x: x[1], reverse=True):
    print(f"{lang}: {score:.2f}%")

🌍 Languages

MirageTVQA covers 45 languages across diverse linguistic families:

Indo-European: English, Spanish, French, Italian, Portuguese, Russian, Czech, Polish, Ukrainian, Romanian, Hindi, Bengali, Marathi, Nepali, Punjabi, Urdu, Sinhala, Persian, Greek, Danish, Norwegian, Swedish

Sino-Tibetan: Chinese (Mandarin), Burmese, Hokkien

Afro-Asiatic: Arabic, Hebrew, Amharic

Austronesian: Indonesian (Formal & Casual), Javanese (Krama & Ngoko), Tagalog, Sundanese, Filipino, Malay

Japonic: Japanese

Koreanic: Korean

Kra-Dai: Thai

Turkic: Turkish, Azerbaijani

Dravidian: Tamil, Telugu

Constructed: Sardinian

📈 Benchmark Results

Best performing models on clean English images:

📄 Citation

If you use MirageTVQA in your research, please cite:

@inproceedings{singh2024mirageTVQA,
  title={Lost in Translation and Noise: A Deep Dive into the Failure Modes of VLMs on Real-World Tables},
  author={Singh, Anshul and Chaudhary, Rohan and Singh, Gagneet and Kumar, Abhay},
  booktitle={EurIPS Workshop on AI for Tabular Data},
  year={2025}
}

License

This dataset is released under the Apache 2.0 License.

Acknowledgements

We thank the creators of the source datasets:

• WikiSQL (Zhong et al., 2017)
• FinQA (Chen et al., 2021)
• ArXiv scientific papers
• GitHub repositories

Links

• Code: GitHub Repository
• Contact: anshulsinghchambial@gmail.com

---