OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding

OCR-MetaReasoning is a controlled benchmark for testing whether multimodal large language models (MLLMs) can organize OCR-grounded visual evidence according to the required reasoning direction—not merely read text from an image.

News and Updates

• [2026/09]: Our paper is now accessible at arXiv.
• [2026/08]: We are delighted that OCR-MetaReasoning has been accepted to EMNLP 2026 Findings!
• [2026/08]: Release the dataset and evaluation script.

Contents

• Overview
• Benchmark at a Glance
• Meta-Reasoning Task
• Taxonomy
• Dataset
• Evaluation
• Results
• Visual Assets
• Quick Start
• Repository Guide
• Paper
• Citation
• Limitations
• Acknowledgements
• License

Overview

Text-rich images encode meaning through words, tables, charts, fields, layout, constraints, legends, and cross-region correspondences. A model can therefore produce a plausible answer while still failing to bind the right visual evidence to the intended reasoning process.

OCR-MetaReasoning evaluates this distinction directly. Every item contains one text-rich image and one question whose dominant bottleneck is explicitly organized as one of three meta-reasoning directions:

• Meta-deduction: apply a visible rule or constraint to a candidate case.
• Meta-induction: infer a hidden regularity from aligned observations and generalize it.
• Meta-abduction: reason backward from an observed result to recover a hidden premise or minimal explanation.

The benchmark reports both final-answer correctness and reasoning-process compliance, making it possible to distinguish “the model answered correctly” from “the model used the intended OCR-grounded reasoning path.”

Benchmark at a Glance

Why this benchmark?

OCR-MetaReasoning is designed around three evaluation requirements:

1. Reasoning direction is explicit. Deduction, induction, and abduction are evaluated as distinct capabilities rather than being collapsed into a generic “reasoning” score.
2. OCR is grounded in visual structure. Relevant evidence can be a word, field, table cell, chart mark, legend, footnote, alignment, or relation between distant regions.
3. Outcome and process are separated. MRMS measures whether the answer is correct; RPCS diagnoses whether the visible solution process follows the intended, grounded reasoning path.

Meta-Reasoning Task

The benchmark uses a hypothesis–rule–observation view of reasoning. Let $H$ denote a hypothesis, candidate state, or hidden premise; $R$ a rule, constraint, mapping, or regularity; and $O$ an observation, result, or consequence grounded in the image.

Expected model response

The evaluator prompts a model to produce a transparent, machine-readable answer structure:

Step 1: ...
Step 2: ...
...
Final Answer: ...

For numeric answers, the final line should contain a number only. For structured answers, it should contain valid JSON. The scorer reads the standalone final-answer line and preserves the preceding steps for RPCS evaluation.

Taxonomy

The benchmark crosses the three meta-reasoning directions with five OCR-object categories. This balanced design prevents performance on a familiar document type or a single reasoning pattern from hiding localized weaknesses.

Balanced distribution

Dataset

Download

The released dataset is hosted on Hugging Face:

Load the hosted dataset with 🤗 Datasets:

from datasets import load_dataset

dataset = load_dataset("GengxuLi123/OCR-MetaReasoning")
print(dataset)

The repository also contains the JSONL benchmark splits and their referenced images:

dataset/
├── meta_deductive/
│   ├── benchmark_meta_deductive.jsonl
│   └── images/
├── meta_inductive/
│   ├── benchmark_meta_inductive.jsonl
│   └── images/
└── meta_abductive/
    ├── benchmark_meta_abductive.jsonl
    └── images/

To load the local JSONL files directly:

from datasets import load_dataset

data_files = {
    "meta_deductive": "dataset/meta_deductive/benchmark_meta_deductive.jsonl",
    "meta_inductive": "dataset/meta_inductive/benchmark_meta_inductive.jsonl",
    "meta_abductive": "dataset/meta_abductive/benchmark_meta_abductive.jsonl",
}

dataset = load_dataset("json", data_files=data_files)

Record schema

Each JSONL record follows the same schema:

Dataset statistics

The released benchmark has complete annotations and automatic scoring specifications for all 1,500 samples. Human verification and a blinded re-annotation study reported 96.0% agreement for meta-reasoning type and 92.7% agreement for OCR-object category.

Evaluation

MRMS: answer-level correctness

The Meta-Reasoning Macro Score (MRMS) is the primary leaderboard metric:

MRMS = (Acc_deductive + Acc_inductive + Acc_abductive) / 3

It averages final-answer performance across the three reasoning directions, so a strong result in one direction cannot dominate the overall score.

Final answers are scored per sample using:

RPCS: process-level compliance

The Reasoning Process Compliance Score (RPCS) is a separate diagnostic over the visible reasoning steps:

RPCS = (capability_match + groundedness + step_completeness + non_hallucination) / 4

RPCS is a visible-process diagnostic, not a claim about hidden internal reasoning. In the reported evaluation, the judge receives the image, question, target labels, reference steps, and model process with the standalone final-answer line removed; the gold answer and MRMS result are withheld.

Results

The following snapshot is reported in the manuscript. Scores are percentage points.

Key observations:

• Across the evaluated models, induction averages 82.5 and abduction 80.1, while deduction is lower at 73.6.
• Layout semantics is the most difficult OCR-object category, averaging 66.9 across models.
• Closed-source models average 82.9 MRMS versus 73.4 for open-source models, a 9.5-point gap.
• RPCS is higher than MRMS for every reported model: plausible, grounded-looking processes can still lead to incorrect final answers.

Visual Assets

The following figures and tables are included as SVG assets under assets/. The source locations use the current manuscript's figure/table numbering and pagination.

Hero example

Benchmark construction

Main results

Process–outcome contrast

Quick Start

1. Install

git clone https://github.com/gengxuli/OCR-MetaReasoning.git
cd OCR-MetaReasoning

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install openai pillow pydantic requests tqdm

Install datasets as well if you want to load the Hugging Face or local JSONL data through 🤗 Datasets:

python -m pip install datasets

2. Configure an OpenAI-compatible endpoint

The inference and RPCS clients use an OpenAI-compatible chat-completions interface. Keep credentials in environment variables or pass them through the command line; do not hard-code keys in source files.

export MODEL_API_KEY="YOUR_API_KEY"
export MODEL_BASE_URL="https://openrouter.ai/v1"
export PYTHONPATH="$(pwd):${PYTHONPATH:-}"

3. Run multimodal inference

--sample_num is applied per selected reasoning type. Thus, 500 with all evaluates up to 500 samples for each of the three splits.

export MODEL_NAME="your-model-name"

python OCR_MetaReasoning/evaluation/openai_infer.py \
  --model_name "${MODEL_NAME}" \
  --base_url "${MODEL_BASE_URL}" \
  --meta_reasoning_types all \
  --sample_num 500 \
  --batch_size 32 \
  --workers 8 \
  --temperature 0.0 \
  --top_p 1.0 \
  --timeout 600

Inference files are written to:

OCR_MetaReasoning/evaluation/result/${MODEL_NAME}_result/

The processor resumes from successful existing records by default. Add --retry_all when a complete rerun is intended.

4. Summarize MRMS and subscores

python OCR_MetaReasoning/evaluation/stats.py \
  --model_name "${MODEL_NAME}"

This writes score_summary.json in the model's result directory and reports scores by reasoning direction, answer type, OCR-object category, and the full 3 × 5 interaction.

5. Evaluate RPCS

RPCS is computed from a completed inference result using a configurable judge model. The default mode includes the original image; use --no_image only for a text-only control.

export JUDGE_MODEL_NAME="your-judge-model-name"

python OCR_MetaReasoning/evaluation/rpcs_judge.py \
  --source_model_name "${MODEL_NAME}" \
  --judge_model_name "${JUDGE_MODEL_NAME}" \
  --base_url "${MODEL_BASE_URL}" \
  --meta_reasoning_types all \
  --workers 8 \
  --batch_size 32 \
  --temperature 0.0 \
  --timeout 600 \
  --max_retries 3

RPCS files are written to:

OCR_MetaReasoning/evaluation/rpcs_result/${MODEL_NAME}_result/

Repository Guide

Paper

OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding
Gengxu Li, Yuan Wu, and Yi Chang

arXiv preprint: arXiv:2608.30678

Citation

If you use OCR-MetaReasoning, please cite:

@misc{li2026ocrmetareasoningbenchmarkevaluatingmetareasoning,
  title         = {OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding},
  author        = {Gengxu Li and Yuan Wu and Yi Chang},
  year          = {2026},
  eprint        = {2608.30678},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2608.30678}
}

Limitations

OCR-MetaReasoning focuses on controlled, single-image, OCR-grounded meta-reasoning. It does not evaluate multi-page evidence chains, cross-image aggregation, retrieval-augmented document reasoning, or interactive clarification. The balanced taxonomy is intended for controlled comparison rather than modeling natural task frequencies. RPCS evaluates visible solution processes and should be interpreted as a diagnostic rather than direct evidence of internal reasoning.

Acknowledgements

This work is supported by the National Key Research and Development Program of China (No. 2023YFF0905400), the National Natural Science Foundation of China (No. U2341229), and the Reform Commission Foundation of Jilin Province (No. 2024C003).

License

See LICENSE for the project license. Please also consult the Hugging Face dataset card and the licenses of any upstream resources before redistributing derived data.