Sarvam Indic OCR Bench

Global benchmarks focus heavily on English document parsing, and to the best of our knowledge there is no Indic OCR benchmark of comparable breadth and rigor. Sarvam Indic OCR Bench fills this gap with 6,909 curated text-block samples drawn from document pages spanning the 19th century to the present, across a wide range of scan quality and content types—including textbooks, newspapers, magazines, and other published material.

The benchmark covers 23 languages: all 22 languages listed in the Eighth Schedule of the Indian Constitution, plus English. Samples are curated at the semantic block level so that models are evaluated on coherent units of text rather than full noisy pages. Each block includes a document image and its corresponding ground-truth transcription.

All ground-truth text has been reviewed twice by human language experts to ensure linguistic accuracy before inclusion in the final benchmark.

Dataset splits

Each record contains:

Language distribution

Samples are balanced as evenly as possible across languages, subject to the availability of high-quality blocks in each language.

Evaluation

We evaluate OCR quality using Character Error Rate (CER) and Word Error Rate (WER). Predictions and ground truth are normalized before scoring (Unicode NFC, newline flattening, quote and dash unification, Indic punctuation standardization, and related formatting rules). Samples with ambiguous ground truth—such as inconsistent colon or slash spacing, ellipsis runs, or transcription artifacts—are excluded from the benchmark.

Word accuracy is reported as:

$$\text{Word Accuracy} = 100 \times (1 - \text{WER})$$

Predictions that exhibit runaway repetition (tail loops) are flagged separately and excluded from valid-sample metrics.

Computing scores

A self-contained scorer, metrics.py, is included in this repo. It is stdlib-only (no third-party packages) and computes CER/WER with the benchmark's content normalization.

1. Get the scorer:

hf download sarvam/indic-ocr-bench metrics.py --repo-type dataset --local-dir .

2. Prepare a predictions JSON — a list of objects with image_name, gt, pred, and (optionally) language for the per-language breakdown:

[
  {"image_name": "sample_id", "gt": "ground truth text", "pred": "model output", "language": "Hindi"}
]

3. Run the scorer:

python metrics.py --input predictions.json --normalize --overwrite

--normalize applies the full content normalization (Unicode NFC/NFKC, newline/whitespace flattening, quote/dash unification, Indic punctuation standardization, and stripping of HTML, quotes, asterisks, bullets/list markers, ZWJ/ZWNJ, and filler rules) so CER/WER reflect genuine word/character errors rather than formatting differences. CER and WER are capped at 1.0 per sample.

The scorer writes a JSON report and prints a summary. Key fields:

Run python metrics.py --help for all options (e.g. --replace-n for newline-only normalization, --output for a custom path).

Usage

from datasets import load_dataset

ds = load_dataset("sarvam/indic-ocr-bench")

# Full benchmark
test = ds["test"]

# Quick per-language subset
small = ds["small_representative"]

print(test[0]["language"], test[0]["gt"][:80])

Citation

If you use this benchmark, please cite:

@misc{sarvam-indic-ocr-bench,
  title={Sarvam Indic OCR Bench},
  author={Sarvam AI},
  year={2026},
  url={https://huggingface.co/datasets/sarvam/indic-ocr-bench}
}