PaperBench: Evaluating AI's Ability to Replicate AI Research

Paper: arXiv:2504.01848
GitHub: openai/preparedness

📋 Dataset Description

PaperBench is a benchmark dataset for evaluating the ability of AI agents to replicate state-of-the-art AI research from scratch. The dataset contains 20 ICML 2024 Spotlight and Oral papers, each decomposed into hierarchical rubrics with clear grading criteria.

Key Statistics

• 20 research papers from ICML 2024
• 8,316 individually gradable tasks across all papers
• Rubrics co-developed with original paper authors for accuracy and realism

🎯 What's Included

Each paper in the dataset contains:

• paper.md - Markdown version of the research paper
• paper.pdf - Original PDF of the paper
• rubric.json - Hierarchical task decomposition with grading criteria
• config.yaml - Paper metadata (ID, title)
• assets/ - Figures and images from the paper
• blacklist.txt - URLs that agents should not access
• addendum.md - Additional context and clarifications

Dataset Manifest

The dataset includes a manifest (parquet file) with metadata for each paper:

🚀 Usage

Loading the Dataset

from datasets import load_dataset

dataset = load_dataset("josancamon/paperbench")

# Access paper metadata
for paper in dataset['train']:
    print(f"{paper['title']}: {paper['num_rubric_tasks']} tasks")
    # the manifest.json only contains metadata of the dataset, for retrieving the papers themselves, look below

Retrieving a paper

from datasets import load_dataset
from huggingface_hub import HfApi

repo_id = "openai/paperbench"
paper_idx = 0
dataset = load_dataset(repo_id)
api = HfApi()
paper_id = dataset["train"][paper_idx]["id"]
downloaded_paths = []
for file in dataset["train"][paper_idx]["reference_files"]:
    local_path = api.hf_hub_download(repo_id=repo_id, filename=file, repo_type="dataset")
    downloaded_paths.append(local_path)

paper_path = Path(downloaded_paths[0]).parent
logger.info(f"Downloaded paper {paper_idx} - {paper_id} successfully to: {paper_path}")

🏗️ Dataset Structure

paperbench/
├── adaptive-pruning/
│   ├── paper.md
│   ├── paper.pdf
│   ├── rubric.json
│   ├── config.yaml
│   ├── blacklist.txt
│   ├── addendum.md
│   └── assets/
│       └── [figures...]
├── all-in-one/
│   └── ...
└── [18 more papers...]

📖 Citation

If you use PaperBench in your research, please cite:

@article{starace2025paperbench,
  title={PaperBench: Evaluating AI's Ability to Replicate AI Research},
  author={Starace, Giulio and Jaffe, Oliver and Sherburn, Dane and Aung, James and Chan, Jun Shern and Maksin, Leon and Dias, Rachel and Mays, Evan and Kinsella, Benjamin and Thompson, Wyatt and Heidecke, Johannes and Glaese, Amelia and Patwardhan, Tejal},
  journal={arXiv preprint arXiv:2504.01848},
  year={2025}
}

📝 License

This dataset is released under the MIT License. Please refer to individual papers for their respective licenses.

🤝 Contributing

For issues or contributions, please visit the GitHub repository.

🔗 Related Work

• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
• SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
• MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

---

Dataset Version: 1.0
Last Updated: October 2025
Contact: For questions about the dataset, please open an issue on GitHub.