StructEval: A Benchmark for Structured Output Evaluation in LLMs

StructEval is a benchmark dataset designed to evaluate the ability of large language models (LLMs) to generate and convert structured outputs across 18 different formats, and 44 types of tasks. It includes both renderable types (e.g., HTML, LaTeX, SVG) and non-renderable types (e.g., JSON, XML, TOML), supporting tasks such as format generation from natural language prompts and format-to-format conversion.

Dataset Summary

Each example in the dataset includes:

• A unique task identifier and name.
• The natural language query.
• Feature-level requirements written in English.
• Input and output format types.
• A complete input query example.
• Metrics for evaluation: raw output keywords, VQA question-answer pairs (for renderable types), and structural rendering flags.

StructEval supports multi-metric evaluation pipelines, including visual rendering checks, VQA scoring via vision-language models, and path-based key validation for structured data.

---

Supported Task Types

• Generation (Text → Format): LLM generates code in a target structured format from a natural language prompt.
• Conversion (Format → Format): LLM converts one structured format into another, e.g., HTML to React or JSON to YAML.

---

Dataset Structure

Features:

Usage Example

from datasets import load_dataset

dataset = load_dataset("your-username/structeval")
example = dataset["train"][0]

print(example["query"])
print(example["VQA"])

Citation

Please cite us with:

@misc{yang2025structeval,
  title={StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs},
  author={Jialin Yang and Dongfu Jiang and Lipeng He and Sherman Siu and Yuxuan Zhang and Disen Liao and Zhuofeng Li and Huaye Zeng and Yiming Jia and Haozhe Wang and Benjamin Schneider and Chi Ruan and Wentao Ma and Zhiheng Lyu and Yifei Wang and Yi Lu and Quy Duc Do and Ziyan Jiang and Ping Nie and Wenhu Chen},
  year={2025},
  eprint={2505.20139},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  doi={10.48550/arXiv.2505.20139}
}