The dataset format is JSONL, with one line per test rule. The structure and field explanations:

{
    "pdf": "docs/chart/report_p41.pdf",   // Relative path to the source document (PDF, JPG, or PNG)
    "category": "chart",                   // Evaluation category
    "id": "unique_rule_id",                // Unique identifier for this test rule
    "type": "chart_data_point",            // Rule type (see below)
    "rule": "{...}",                       // JSON-encoded rule payload with evaluation parameters
    "page": null,                          // Page number (1-indexed), used by layout rules
    "expected_markdown": null,             // Ground-truth HTML/markdown, used by table rules
    "tags": ["need_estimate"]              // Document-level tags for filtering and grouping
}

Tags by category:

• chart: need_estimate (value requires visual estimation), 3d_chart (3D chart rendering)
• table: difficulty (easy, hard)
• text_content / text_formatting: difficulty (easy, hard) and document type (dense, sparse, simple, multicolumns, ocr, multilang, misc, handwritting)
• layout: difficulty (easy, hard)

Rule types by category:

• chart: chart_data_point — a spot-check data point specifying a numerical value and one or more labels (series name, x-axis category) that should be locatable in the parser's table output, with a configurable tolerance.
• table: expected_markdown — ground-truth HTML table structure. Evaluation treats tables as bags of records (rows keyed by column headers).
• layout: layout (bounding box + semantic class + content + reading order index), order (pairwise reading order assertion).
• text_content: missing_word_percent, unexpected_word_percent, too_many_word_occurence_percent, missing_sentence_percent, unexpected_sentence_percent, too_many_sentence_occurence_percent, bag_of_digit_percent, order, missing_specific_word, missing_specific_sentence, is_footer, is_header
• text_formatting: is_bold, is_italic, is_underline, is_strikeout, is_mark, is_sup, is_sub, is_title, title_hierarchy_percent, is_latex, is_code_block

Chart rule type — chart_data_point:

Each rule specifies an expected numerical value and one or more labels (series name, x-axis category, chart title). A data point is verified if its value and all associated labels can be located in the parser's table output. Evaluation is insensitive to table orientation (rows and columns can be swapped) and tolerant of numeric formatting differences (currency symbols, unit suffixes, thousands separators). Each data point includes a configurable tolerance since exact value retrieval from charts is often imprecise.

chart_data_point    # Spot-check data point: value + labels matched against parser's table output
                    # Rule fields: labels (list), value (string), max_diffs (int), normalize_numbers (bool)

Table — expected_markdown:

Each rule provides a ground-truth HTML table. Evaluation uses the TableRecordMatch metric, which treats a table as a bag of records: each row is a record whose cell values are keyed by their column headers. Ground-truth records are matched to predicted records, and each matched pair is scored by binary cell-level agreement. TableRecordMatch is insensitive to column and row order (which don't alter key-value relationships), while dropped or transposed headers cause large mismatches and are penalized accordingly.

expected_markdown   # Ground-truth HTML table for TableRecordMatch evaluation
                    # Rule fields: {} (ground truth stored in expected_markdown field)

Text Content rule types measure whether the parser faithfully reproduces textual content:

# Text correctness — omissions and hallucinations
missing_word_percent              # Fraction of ground-truth words missing from output
unexpected_word_percent           # Fraction of output words not in ground truth (hallucinations)
too_many_word_occurence_percent   # Excess word duplications
missing_sentence_percent          # Fraction of ground-truth sentences missing
unexpected_sentence_percent       # Fraction of output sentences not in ground truth
too_many_sentence_occurence_percent  # Excess sentence duplications
bag_of_digit_percent              # Digit frequency distribution match (catches OCR errors like 6→8)
missing_specific_word             # Binary: specific word present or absent
missing_specific_sentence         # Binary: specific sentence present or absent

# Structural
order                             # Pairwise reading order assertion (before/after)
is_footer                         # Footer detection
is_header                         # Header detection

Text Formatting rule types verify preservation of semantically meaningful formatting:

# Text styling
is_bold                   # Bold formatting preserved
is_italic                 # Italic formatting preserved
is_underline              # Underline formatting preserved
is_strikeout              # Strikethrough preserved (marks superseded content)
is_mark                   # Highlight/mark preserved
is_sup                    # Superscript preserved (footnotes, exponents)
is_sub                    # Subscript preserved (chemical formulae)

# Document structure
is_title                  # Text appears as heading at correct level
title_hierarchy_percent   # Title parent-child hierarchy score

# Special content
is_latex                  # Mathematical formula in LaTeX notation
is_code_block             # Fenced code block with language annotation

Layout rule types evaluate visual grounding:

layout    # Element annotation: bounding box (normalized [0,1]),
          # semantic class (Text, Table, Picture, Page-Header, Page-Footer),
          # content association, and reading order index
order     # Layout-level reading order assertion

Chart documents (568 pages) — bar, line, pie, and compound charts from corporate reports, financial filings, and government publications. The dataset ensures diversity across charts with/without explicit value labels, discrete and continuous series, varying data density, and single vs. multi-chart pages.

Table documents (503 pages) — sourced primarily from insurance filings (SERFF), public financial documents, and government reports. Tables remain embedded in their original PDF pages, preserving the full visual context. The dataset includes merged cells, hierarchical headers, spanning rows, and multi-page tables.

Text documents (508 pages, shared by Content Faithfulness and Semantic Formatting) — one page per document, categorized by tag:

Layout documents (500 pages) — single-column, multi-column, and complex layouts with mixed media (text, images, tables, charts). Includes PDF, JPG, and PNG inputs. Evaluation uses a compact label set: Text, Table, Picture, Page-Header, and Page-Footer.