VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents

VAREX (VARied-schema EXtraction) is a benchmark for evaluating multimodal foundation models on structured data extraction from government forms. It comprises 1,777 documents with 1,771 unique schemas across three structural categories, each provided in four input modalities. Ground truth is deterministic — generated via a Reverse Annotation pipeline that programmatically fills PDF templates with synthetic values, validated through three-phase quality assurance achieving ~98.5% field-level accuracy.

Paper: arXiv:2603.15118

Evaluation code & scoring: github.com/udibarzi/varex-bench

Quick Start

from datasets import load_dataset
import json

ds = load_dataset("ibm-research/VAREX", split="benchmark")

doc = ds[0]
print(doc["doc_id"])        # e.g., "1044"
print(doc["split"])         # "Flat", "Nested", or "Table"

schema = json.loads(doc["schema"])
gt = json.loads(doc["ground_truth"])
image = doc["image"]        # PIL Image, 200 DPI
text = doc["text_layout"]   # Spatial text with layout

Columns

Input Modalities

Document Splits

PDF Files

Original filled PDFs are available in the pdfs/ directory of this repository. Each filename corresponds to the doc_id column (e.g., doc_id "1044" → pdfs/1044.pdf). These allow researchers to apply their own text extraction or parsing pipelines.

Scoring

Evaluation code, scoring scripts, and field exclusion lists are maintained at: github.com/udibarzi/varex-bench

The benchmark uses Exact Match (EM) as the primary metric with order-invariant array matching via the Hungarian algorithm. 610 field-level exclusions are applied at scoring time for fields with known ground truth issues.

Citation

@inproceedings{varex2026,
  title   = {VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents},
  author  = {Barzelay, Udi and Azulai, Ophir and Shapira, Inbar and Friedman, Idan and Abo Dahood, Foad and Lee, Madison and Daniels, Abraham},
  year    = {2026}
}

License

Community Data License Agreement – Permissive, Version 2.0