PaperScope Dataset Card

Dataset Description

PaperScope is a multi-modal, multi-document benchmark designed for evaluating agentic deep research capabilities across massive scientific papers. Built upon a knowledge graph of over 2,000 AI papers from top-tier venues (NeurIPS, ICLR, ICML, etc.), this dataset enables comprehensive evaluation of multi-modal large language models (MLLMs) on tasks requiring reasoning, retrieval, summarization, and problem-solving across multiple scientific documents.

Dataset Summary

⚠️ Important Notice: The test set is intentionally NOT publicly released to prevent data leakage and maintain the integrity of benchmark evaluations. Researchers who wish to evaluate on the test set should contact the authors for blind evaluation.

Corpus Data Notice

📦 Due to the large size of the full document corpus, we only release paper identifiers (OpenReview Paper IDs or paper titles) in the public version. The referenced papers can be accessed through:

• OpenReview: https://openreview.net/pdf?id={paper_id}
• Paper titles can be used to search on OpenReview, arXiv, or Semantic Scholar

Dataset Structure

Data Files

data/
├── camera_data/                    # Processed evaluation data
│   ├── corpus_1.jsonl              # Paper identifiers (500 papers, title + OpenReview ID)
│   ├── corpus_2.jsonl              # Paper identifiers (2000 papers, title + OpenReview ID)
│   ├── induction_corpus_1.jsonl    # Paper retrieval/induction tasks
│   ├── reasoning_corpus_1.jsonl    # Multi-hop reasoning tasks
│   ├── solution_corpus_1.jsonl     # Problem-solving tasks
│   └── summary_corpus_1.jsonl      # Summarization tasks
└── README.md                       # This file

Paper Corpus Files

The paper corpus files contain paper identifiers that can be used to access the original papers from OpenReview.

corpus_1.jsonl (500 papers)

Example:

{"title": "Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning", "id": "09iOdaeOzp"}

corpus_2.jsonl (2000 papers)

Same structure as corpus_1.jsonl, containing a larger set of 2000 papers from the knowledge graph.

Accessing Papers:

# Access paper PDF via OpenReview
paper_id = "09iOdaeOzp"
pdf_url = f"https://openreview.net/pdf?id={paper_id}"

---

Task Types

1. Induction Corpus (induction_corpus_1.jsonl)

Paper retrieval tasks where models must identify relevant papers given a complex, multi-faceted research query.

Example:

{
  "question": "Help me find papers that collectively address training efficiency...",
  "answer": [
    {"title": "Memory Efficient Optimizers with 4-bit States", "total_entities": 4892, ...},
    {"title": "DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining", ...}
  ],
  "type": "explicit",
  "common_entities": ["SGD", "fine-tuning", "Transformers", ...]
}

2. Reasoning Corpus (reasoning_corpus_1.jsonl)

Multi-hop reasoning tasks requiring evidence integration from multiple papers.

Subtask Types:

• Figure-table-chart comparison - Compare visual elements across papers
• Algorithm reasoning - Analyze algorithmic procedures
• Formula reasoning - Mathematical formula analysis
• Figure-table-chart reasoning - Interpret figures/tables
• Paper analysis - Comprehensive paper comparison

3. Solution Corpus (solution_corpus_1.jsonl)

Problem-solving tasks requiring synthesis of methods from multiple papers.

4. Summary Corpus (summary_corpus_1.jsonl)

Summarization tasks for multi-paper synthesis.

Prompt Types:

• trend - Development trend analysis
• gap - Research gap/limitation analysis
• results_comparison - Experimental results comparison

Dataset Statistics

Total Samples: 1,360

Data Collection

Source Papers

• Venues: NeurIPS, ICLR, ICML, and other top-tier AI conferences
• Years: 2023-2025
• Total Papers: 2,000+ papers in the knowledge graph
• Paper Access: Via OpenReview (paper IDs provided)

Knowledge Graph Construction

Papers are processed to extract:

• Classification tags
• Research background
• Key contributions
• Methodology details
• Experimental results
• Figures, tables, and formulas

QA Generation

Questions are generated through:

1. Semantically dense evidence construction - Integrating related key information nodes
2. Optimized random-walk article selection - Ensuring diverse, coherent paper sets
3. Multi-task prompt engineering - Creating varied evaluation scenarios

Usage

Loading the Dataset

import json

def load_jsonl(filepath):
    data = []
    with open(filepath, 'r', encoding='utf-8') as f:
        for line in f:
            data.append(json.loads(line.strip()))
    return data

# Load different corpora
induction_data = load_jsonl('data/camera_data/induction_corpus_1.jsonl')
reasoning_data = load_jsonl('data/camera_data/reasoning_corpus_1.jsonl')
solution_data = load_jsonl('data/camera_data/solution_corpus_1.jsonl')
summary_data = load_jsonl('data/camera_data/summary_corpus_1.jsonl')

Accessing Source Papers

# From reasoning corpus
sample = reasoning_data[0]
paper_links = sample['answer_docs_links']

# Access papers via OpenReview
for link in paper_links:
    paper_id = link.split('id=')[1]
    print(f"Paper ID: {paper_id}")
    print(f"PDF URL: https://openreview.net/pdf?id={paper_id}")

Evaluation

Metrics

Evaluation Protocol

1. Retrieval-Augmented Generation (RAG): Models retrieve relevant papers before answering
2. Direct Generation: Models generate answers from provided paper references
3. Agentic Evaluation: Models autonomously search, retrieve, and synthesize

Limitations and Ethical Considerations

Limitations

• Dataset focuses on AI/ML domain papers
• English-only content
• Papers limited to open-access venues (primarily OpenReview)
• Knowledge graph reflects snapshot of papers at construction time

Ethical Considerations

• All source papers are publicly accessible
• No personally identifiable information
• Dataset intended for research purposes only
• Users should cite original papers when applicable

Citation

If you use this dataset, please cite:

@article{paperscope2025,
  title={PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers},
  author={Lei Xiong, Huaying Yuan, Zheng Liu, Zhicheng Dou, Zhao Cao},
  year={2026}
}

License

This dataset is released under CC BY-NC 4.0.

• ✅ Academic research use
• ✅ Non-commercial applications
• ❌ Commercial use without permission

Contact

For questions, test set evaluation requests, or issues:

• Open an issue on GitHub
• Contact the authors via email (see paper)

---

Last Updated: January 2026