ReportBench — Dataset Card

Overview

ReportBench is a comprehensive benchmark for evaluating the factual quality and citation behavior of Deep Research agents. Leveraging expert-authored survey papers as ground truth, ReportBench reverse-engineers domain-specific prompts and provides automated tools to assess both cited and non-cited content. [GitHub] / [Paper]

ReportBench addresses this need by:

• Leveraging expert surveys: Uses high-quality, peer-reviewed survey papers from arXiv as gold-standard references.
• Reverse-engineering prompts: Generates task-specific prompts matching each survey’s scope, methods, and temporal constraints.
• Automated validation: Employs a dual-path evaluation to verify citation consistency and factual correctness of non-cited statements.

---

Benchmark Construction

The dataset construction pipeline consists of four phases:

1. Survey Paper Identification

   • Start from an arXiv metadata snapshot (post-2020).
   • Filter titles/abstracts for “survey” or “review” and confirm publication status via metadata and LLMs classification.
   • Retain 600 high-quality, peer-reviewed survey papers.

2. Fine-Grained Reference Extraction

   • Download and parse LaTeX sources to extract all in-text citation commands.
   • Build a gold-standard set of references mirroring the true citation pattern of each survey.

3. Prompt Generation

   • Reverse-engineer three levels of prompts (sentence, paragraph, detail-rich) via LLMs.
   • Enforce temporal constraints matching each paper’s publication cutoff.
   • Add explicit instructions to avoid citing the original survey itself.

4. Application Domain Distribution

   • Classify surveys into ten domains using LLMs.
   • Perform downsampling while adjusting the distribution to achieve balance, and sample one of the three prompt types to form a 100-task benchmark.

---

Evaluation Framework

ReportBench’s evaluation workflow consists of two complementary validation procedures:

1. Content Quality

• URL Extraction: Extract all URL citations from the report, including from the base model and Deep Research outputs.
• Normalization and Retrieval: Normalize and deduplicate URLs, then retrieve the content of each web page.
• Document Type Classification: Use an LLM to determine whether each URL corresponds to a scholarly article and extract its title if applicable.
• Title Matching: Compare the extracted titles against ground-truth references from the expert-authored report and compute an overlap ratio.

2.1. Cited Statements

• Statement Extraction: Identify all sentences in a generated report containing explicit citations.
• Source Retrieval: Scrape the full text of each cited source.
• Semantic Matching: Use an LLM to locate supporting passages and verify consistency.
• Scoring: Compute a citation alignment score for each report.

2.2. Non-Cited Statements

• Statement Extraction: Extract factual claims without citations, filtering out common-sense content.
• Web-Connected Fact Checking: Query multiple web-connected LLMs (Gemini Pro and Flash) to independently verify each claim.
• Voting Mechanism: Aggregate judgments via majority vote to compute factual accuracy.

---

Evaluation Results

We evaluated two Deep Research products alongside their corresponding base LLMs using the ReportBench benchmark. Table 1 summarizes precision, recall, average references per report, citation match rate, cited statement count, non-cited factual accuracy, and non-cited statement count.

Table 1. Performance metrics of Deep Research products and their base models.

Product-Level

• OpenAI Deep Research: Highest precision (0.385) and citation match rate (78.87%), indicating focused and accurate retrieval with fewer references.
• Gemini Deep Research: Generates many more citations (32.42 vs. 9.89) but yields only marginal recall gain, suggesting over-generation without proportional coverage benefits.

Model-Level

• OpenAI vs. o3: Comparable retrieval metrics, but Deep Research produces far more cited statements (88.2 vs. 16.16) and non-cited statements (38.9 vs. 11.51), and achieves much higher alignment (78.87% vs. 31.43%) and accuracy (95.83% vs. 82.22%).
• Gemini Deep Research vs. gemini-2.5-pro: Trades off precision (0.145 vs. 0.269) for higher recall and citation volume, while maintaining strong alignment (72.94% vs. 59.24%) but slightly lower non-cited statement accuracy.
• claude4-sonnet: Most balanced baseline—moderate precision (0.337), recall (0.021), citation consistency (73.67%), and non-cited statement accuracy (92.64%).

---

Format & Example record

Format: JSON Lines (.jsonl) — one JSON object per line. Each object describes a task and includes a ground_truth array containing reference objects (bibs).

{"arxiv_id":"2312.04861","submitter":"Shanliang Yao","authors":"Shanliang Yao, Runwei Guan, Zitian Peng, Chenhang Xu, Yilu Shi, Weiping Ding, Eng Gee Lim, Yong Yue, Hyungjoon Seo, Ka Lok Man, Jieming Ma, Xiaohui Zhu, Yutao Yue","title":"Exploring Radar Data Representations in Autonomous Driving: A Comprehensive Review","comments":"Accepted by TITS","journal-ref":"IEEE Transactions on Intelligent Transportation Systems 2025","doi":"10.1109\/TITS.2025.3554781","report-no":null,"categories":"cs.CV cs.AI","license":"http:\/\/arxiv.org\/licenses\/nonexclusive-distrib\/1.0\/","abstract":"With the rapid ... radar.","versions":[{"version":"v1","created":"Fri, 8 Dec 2023 06:31:19 GMT"},{"version":"v2","created":"Fri, 19 Apr 2024 08:55:34 GMT"},{"version":"v3","created":"Mon, 21 Apr 2025 08:37:24 GMT"}],"update_date":"2025-04-22","authors_parsed":[["Yao","Shanliang"],["Guan","Runwei"],["Peng","Zitian"],["Xu","Chenhang"],["Shi","Yilu"],["Ding","Weiping"],["Lim","Eng Gee"],["Yue","Yong"],["Seo","Hyungjoon"],["Man","Ka Lok"],["Ma","Jieming"],["Zhu","Xiaohui"],["Yue","Yutao"]],"application_domain":"Transportation and Intelligent Mobility","prompt":"Please help me research the academic advancements in different radar data representation methods in the field of autonomous driving, and ensure only papers published before April 2025 are referenced.","ground_truth":[{"bib_id":"zheng2022tj4dradset","title":"TJ4DRadSet: A 4D radar dataset for autonomous driving","author":"Zheng, Lianqing and Ma, Zhixiong and Zhu, Xichan and Tan, Bin and Li, Sen and Long, Kai and Sun, Weiqi and Chen, Sihan and Zhang, Lu and Wan, Mengyue and others","meta_info":{"organization":"IEEE","year":"2022","pages":"493--498","booktitle":"2022 IEEE 25th international conference on intelligent transportation systems (ITSC)"}},...,]}
{"arxiv_id":"2308.06419","submitter":"Mahsa Golchoubian","authors":"Mahsa Golchoubian, Moojan Ghafurian, Kerstin Dautenhahn, Nasser Lashgarian Azad","title":"Pedestrian Trajectory Prediction in Pedestrian-Vehicle Mixed Environments: A Systematic Review","comments":"Published in IEEE Transactions on Intelligent Transportation Systems","journal-ref":null,"doi":"10.1109\/TITS.2023.3291196","report-no":null,"categories":"cs.RO cs.AI cs.HC cs.LG cs.SY eess.SY","license":"http:\/\/arxiv.org\/licenses\/nonexclusive-distrib\/1.0\/","abstract":"Planning an ... environments are discussed.","versions":[{"version":"v1","created":"Fri, 11 Aug 2023 23:58:51 GMT"}],"update_date":"2023-08-15","authors_parsed":[["Golchoubian","Mahsa"],["Ghafurian","Moojan"],["Dautenhahn","Kerstin"],["Azad","Nasser Lashgarian"]],"application_domain":"Transportation and Intelligent Mobility","prompt":"Please help me summarize the research status in the field of pedestrian trajectory prediction in unstructured environments with human-vehicle interactions prior to August 2023.","ground_truth":[{"bib_id":"helbing1995social","title":"Social force model for pedestrian dynamics","author":"Helbing, Dirk and Molnar, Peter","meta_info":{"publisher":"APS","year":"1995","pages":"4282--4286","number":"5","volume":"51","journal":"Physical review E"}},{"bib_id":"yang2018crowd","title":"Crowd motion detection and prediction for transportation efficiency in shared spaces","author":"Yang, Dongfang and Maroli, John M and Li, Linhui and El-Shaer, Menna and Jabr, Bander A and Redmill, Keith and \u00d6zguner, F\u00fcsun and \u00d6zguner, \u00dcmit","meta_info":{"organization":"IEEE","year":"2018","pages":"1--6","booktitle":"2018 IEEE International Science of Smart City Operations and Platforms Engineering in Partnership with Global City Teams Challenge (SCOPE-GCTC)"}}, ...,]}

---

Limitations

• Ground truth is derived from survey authors’ citations — surveys have authorship and coverage bias; treat ground truth as representing the survey’s citation choices rather than an absolute canonical list.
• Respect publisher copyrights; the dataset contains metadata and reference records rather than full-text copies of copyrighted works.

---

Citation

@misc{li2025reportbenchevaluatingdeepresearch,
      title={ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks}, 
      author={Minghao Li and Ying Zeng and Zhihao Cheng and Cong Ma and Kai Jia},
      year={2025},
      eprint={2508.15804},
      archivePrefix={arXiv},