Table Metric Study

Meta-evaluation of table extraction metrics against human judgment, accompanying the paper:

Beyond String Matching: Semantic Evaluation of PDF Table Extraction

This repository provides implementations of rule-based table metrics (TEDS, GriTS, SCORE), LLM-as-a-judge scoring, a human evaluation interface, and the correlation analysis used to validate that LLM-based evaluation substantially outperforms rule-based metrics in agreement with human judgment.

Results

The dataset includes over 1,500 human quality ratings on 518 table pairs. The correlation analysis shows that LLM-based judges achieve substantially higher agreement with human judgment than rule-based metrics:

Correlation of each metric with the averaged human scores (three annotators per table pair):

LLM judge costs are based on OpenRouter pricing as of 2026-04-24. The TabXEval cost was measured on a 5-pair sample (two gpt-4o calls per pair) at OpenRouter pricing as of 2026-07-04.

Notably, TabXEval — a two-phase LLM pipeline (alignment + fine-grained comparison, run with its original prompts and gpt-4o judge) — correlates worse with human judgment than the rule-based metrics here, despite costing more than any single-prompt LLM judge. 60% of pairs score exactly 1.0, including 20 pairs with an average human rating below 5: its alignment step tends to silently repair structurally broken extractions (e.g. re-splitting merged rows), so the subsequent comparison finds no differences left to penalize.

Prompt sensitivity

Same 518 extractions, three prompt variants:

tuned — the prompt used for the main results. Response schema {errors[], score}.

You are a strict table evaluator. Your task is to determine if the extracted table correctly represents the ground truth table, focusing on content accuracy, structural preservation, and information completeness. The extracted table was parsed from the rendered table. Disregard LaTeX-specific elements in the ground truth (e.g., comments, styling commands, font formatting) that have no effect on content or structure.

Ground Truth Table (LaTeX):
{gt_table}

Extracted Table:
{extracted_table}

Evaluate the extracted table using the following criteria:
1. Content accuracy: Are all cell values, headers, and data correctly preserved?
2. Structure preservation: Are all rows and columns present, and can each value be unambiguously mapped to its row/column headers? Broken or ambiguous associations count as errors.

Note: Different output formats (markdown, HTML, plain text) are acceptable as long as no information is lost. Apply this key test: Could a reader who sees ONLY the extracted table — without access to the ground truth — unambiguously reconstruct every cell-to-header mapping and all content of the original table? If not, consider the parsing as failed and assign a low score.

First, enumerate up to 5 of the most significant errors and ambiguities found. Then assign a score from 0 to 10, where 10 is a perfect match.

tuned_no_cot — identical to tuned with the final sentence replaced by "Assign a score from 0 to 10, where 10 is a perfect match." and response schema reduced to {score}. Isolates the CoT scaffold.

naive — "Rate how well the extracted table matches the ground truth table on a scale from 0 to 10." Response schema {score}.

Both deltas are taken relative to the naive baseline: Δ content = tuned_no_cot − naive, Δ full = tuned − naive.

The CoT scaffold and engineered content interact: gemma-4-26b and deepseek-v3.2 show negative Δ content but positive Δ full, while mistral-small shows the inverse. Of 24 (model × variant) cells, only deepseek-v3.2/tuned_no_cot (0.671) falls below the best rule-based metric (GriTS-Con 0.701).

Project Structure

Reproducing

Requires Python 3.12+ and uv. All scripts can be run via uv run python <script>.py.

uv sync

System dependencies for rule-based metrics and human evaluation UI:

• pdflatex and pdftoppm (e.g., via TeX Live)
• latexmlc (for LaTeX-to-HTML normalization)

LLM scoring requires an OpenRouter API key (export OPENROUTER_API_KEY=...).

TabXEval (Pancholi et al., Findings of ACL 2025) is included as an additional LLM-based baseline via compute_tabxeval_scores.py. It aligns ground truth and extraction with a rule-based fuzzy merge plus an LLM alignment call (TabAlign), classifies every non-exact cell difference with a second LLM call (TabCompare), and aggregates the classified differences into a penalty-based score in [0, 1]. The implementation under scorers/tabxeval/ is vendored from the official repository (MIT License) with the original prompts and gpt-4o judge (via OpenRouter), and the scoring logic ported from its eval_scores.ipynb.

Data Format

Each entry in all_tables.json pairs a ground truth table with its parser extractions, metric scores, and human ratings:

{
  "gt_id": "000_00",
  "gt_table": "\\begin{tabular}...",
  "gt_table_html": "<table>...</table>",
  "complexity": "simple | moderate | complex",
  "extractions": [
    {
      "parser": "gemini_3_flash",
      "extracted_table": "...",
      "metrics": { "teds": 0.91, "grits_top": 0.89, "grits_con": 0.87, ... },
      "llm_scores": [
        { "judge_model": "google/gemini-3-flash-preview", "score": 9, "errors": [...] }
      ],
      "tabxeval": { "judge_model": "openai/gpt-4o", "score": 0.86 },
      "human_scores": [8, 8, 7]
    }
  ]
}