OCR-MetaReasoning is a controlled benchmark for testing whether multimodal large language models (MLLMs) can organize OCR-grounded visual evidence according to the required reasoning direction, rather than merely copy text from an image.

Overview

Text-rich images encode meaning through words, tables, charts, fields, layout, constraints, legends, and cross-region correspondences. OCR-MetaReasoning evaluates the ability to use these signals in the reasoning direction required by a question.

Each item contains one text-rich image, one question, a canonical answer, an answer format and scoring rule, and a reference reasoning path. The release is balanced across three meta-reasoning types and five OCR-object categories, enabling controlled analysis of both answer correctness and visible reasoning-process compliance.

_{Benchmark construction pipeline. See the paper for the full protocol.}

Benchmark at a Glance

Task and Taxonomy

Meta-reasoning types

The label is assigned according to the dominant bottleneck needed to recover the target variable. A sample may still contain local arithmetic, lookup, comparison, or OCR operations.

OCR-object categories

Every combination of one reasoning type and one OCR-object category contains 100 samples.

Dataset

Configs and layout

The dataset provides three independent Hugging Face configs. Each config exposes a train split with 500 annotated samples. The split name reflects the released file structure; it is the complete annotated benchmark subset, not a predefined train/test partition.

The extracted project layout is:

dataset/
├── meta_deductive/
│   ├── benchmark_meta_deductive.jsonl
│   └── images/
├── meta_inductive/
│   ├── benchmark_meta_inductive.jsonl
│   └── images/
└── meta_abductive/
    ├── benchmark_meta_abductive.jsonl
    └── images/

The three subset directories are independent. Image paths in each JSONL file are relative to that subset, and the downloaded/extracted layout mirrors the repository dataset/ directory.

Load from Hugging Face

pip install -U datasets

from datasets import load_dataset

repo_id = "GengxuLi123/OCR-MetaReasoning"
configs = ["meta_deductive", "meta_inductive", "meta_abductive"]

subsets = {
    config: load_dataset(repo_id, config, split="train")
    for config in configs
}

sample = subsets["meta_deductive"][0]
print(sample["question"])
print(sample["answer_type"], sample["metric"])
sample["image"].save("example.png")

For benchmark evaluation, use only image and question as model inputs. Keep answer and reasoning_steps hidden from the model.

Load the local mirror

from datasets import load_dataset

local_files = {
    "meta_deductive": "dataset/meta_deductive/benchmark_meta_deductive.jsonl",
    "meta_inductive": "dataset/meta_inductive/benchmark_meta_inductive.jsonl",
    "meta_abductive": "dataset/meta_abductive/benchmark_meta_abductive.jsonl",
}

local_subsets = {
    config: load_dataset("json", data_files={"train": path}, split="train")
    for config, path in local_files.items()
}

Record schema

Answer distribution

Evaluation

Output format

Models are prompted to return numbered reasoning steps followed by a standalone final-answer line:

Step 1: ...
Step 2: ...
...
Final Answer: ...

The released evaluator extracts only the final-answer line for answer scoring and retains the preceding visible process for RPCS evaluation.

MRMS: answer-level correctness

The primary metric is the Meta-Reasoning Macro Score (MRMS):

MRMS = (Acc_deductive + Acc_inductive + Acc_abductive) / 3

Each Acc is the mean per-sample score for one reasoning subset. The macro average prevents performance on one reasoning direction from hiding weakness in another. MRMS is normalized to [0, 1]; paper tables report percentage points.

RPCS: process-level compliance

The Reasoning Process Compliance Score (RPCS) is a separate diagnostic over the visible reasoning process:

RPCS = (capability_match + groundedness + step_completeness + non_hallucination) / 4

RPCS evaluates visible rationales and should not be interpreted as a direct measurement of hidden internal reasoning. The paper's judge protocol removes the standalone final-answer line and withholds the gold answer, predicted answer, and MRMS score.

Reported reference results

The following results are reported in the paper. Scores are percentage points; MRMS is deterministic answer scoring and RPCS is the judge-based process diagnostic.

Data Construction and Quality

The benchmark was constructed from public OCR-oriented seeds covering receipts, charts, forms, policy-like documents, posters, webpages, and infographics. Source collections include CORD, WildReceipt, ChartQA, ChartXiv, FUNSD, DocVQA, InfoVQA, and TextVQA. MLLMs assisted with sample synthesis and controlled image edits; human verification determined whether a candidate was retained.

Three PhD students with MLLM research experience checked each retained item against five requirements: image-grounded primary evidence, at least two required evidence points, a unique or uniquely minimal answer, a label matching the dominant reasoning bottleneck, and a compatible automatic scorer.

The final release contains 1,299 unmodified images and 201 edited/reconstructed images. All edited/reconstructed images are in the meta_abductive subset.

Limitations and Responsible Use

• The benchmark is limited to single-image, text-rich reasoning. It does not evaluate multi-page evidence chains, cross-image aggregation, retrieval-augmented document reasoning, or interactive clarification.
• The 3 x 5 balance is designed for controlled comparison and does not represent natural task frequencies.
• MLLM-assisted synthesis and controlled edits may differ from fully natural documents. The reasoning label denotes the dominant bottleneck, although a sample can contain other local operations.
• The repository declares the MIT license. Some images are derived from public OCR-oriented source collections or controlled variants; users should review the applicable upstream terms before redistributing derived data.

Paper and Code

• Paper: OCR-MetaReasoning Benchmark
• Code and evaluation scripts: gengxuli/OCR-MetaReasoning

Citation

@misc{li2026ocrmetareasoningbenchmarkevaluatingmetareasoning,
  title         = {OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding},
  author        = {Gengxu Li and Yuan Wu and Yi Chang},
  year          = {2026},
  eprint        = {2608.30678},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2608.30678}
}

Acknowledgements

This work is supported by the National Key Research and Development Program of China (No. 2023YFF0905400), the National Natural Science Foundation of China (No. U2341229), and the Reform Commission Foundation of Jilin Province (No. 2024C003).