❗ (Jan 19, 2026) We are currently working on an update that will improve the robustness and solvability of our dataset tasks. Please stay tuned! ❗
EXP-Bench is a novel benchmark designed to systematically evaluate AI agents on complete research experiments sourced from influential AI publications. Given a research question and incomplete starter code, EXP-Bench challenges AI agents to formulate hypotheses, design and implement experimental procedures, execute them, and analyze results. This dataset curates 461 AI research tasks from 51 top-tier AI research papers.
| Github | 🏆 Leaderboard TBD | 📖 Paper |
We have a pipeline that automates the evaluation process, consisting of generating output and judging. Please refer to our Github page for instructions.
[TBD]
To be released soon..
If you encounter any errors in the dataset, please report the row number on our issue page, and we will address it accordingly. Our team is dedicated to maintaining this dataset over the long term to ensure its quality.
The verification configuration contains 22 task-level records: 7 MATCH, 7 PARTIAL_MATCH, 7 NO_CONCLUSION and 1 DIFFERENT. The original default configuration and its 11 columns remain unchanged.
The verification table uses two existing task identifiers and six additional fields:
| Field | Purpose |
|---|---|
paper_id, task_index | Link to the original benchmark task |
verification_status | Recorded task-level verdict |
verification_setup | Experimental conditions, including data or environment |
verification_result | Reported measurements or stage outcomes |
verification_scope | What the result supports under those conditions |
verification_source | Link to the recorded verification summary |
verification_git_diff | Unified diff for the corresponding experiment, when available |
The results documentation describes five quantitative examples covering forecasting, reasoning, multimodal retrieval, reinforcement learning and neural reconstruction. The measurements and their experimental conditions provide references for assessing research-task execution and the conclusions supported by the pipeline.
from datasets import load_dataset
results = load_dataset("Just-Curieous/EXP-Bench", "verification", split="audit")
The first 21 verification records refer to the May 20, 2026 progress snapshot. The additional 18439/task1 record reports the October 7, 2026 native-calibration trial. The original default configuration contains all 461 benchmark tasks.
verification_git_diff is reserved for the unified diff corresponding to the reported experiment. The 18439/task1 value contains the complete experimental-code patch and source/base/launch instructions; two machine-specific interpreter strings in logging metadata are replaced with python3 for publication. All 21 historical values remain empty because their corresponding experimental-code patches have not been recovered from the saved artifacts. Empty values do not mean that no code changes were made.
A usable patch should identify its source repository and base commit in comment headers, followed by the unified diff. Readers can check out that base commit, save the field as experiment.patch, and use git apply --check experiment.patch followed by git apply experiment.patch. Execution commands and data preparation must also be supplied before a complete reproduction is possible.
❗ (Jan 19, 2026) We are currently working on an update that will improve the robustness and solvability of our dataset tasks. Please stay tuned! ❗
EXP-Bench is a novel benchmark designed to systematically evaluate AI agents on complete research experiments sourced from influential AI publications. Given a research question and incomplete starter code, EXP-Bench challenges AI agents to formulate hypotheses, design and implement experimental procedures, execute them, and analyze results. This dataset curates 461 AI research tasks from 51 top-tier AI research papers.
| Github | 🏆 Leaderboard TBD | 📖 Paper |
We have a pipeline that automates the evaluation process, consisting of generating output and judging. Please refer to our Github page for instructions.
[TBD]
To be released soon..
If you encounter any errors in the dataset, please report the row number on our issue page, and we will address it accordingly. Our team is dedicated to maintaining this dataset over the long term to ensure its quality.
The verification configuration contains 22 task-level records: 7 MATCH, 7 PARTIAL_MATCH, 7 NO_CONCLUSION and 1 DIFFERENT. The original default configuration and its 11 columns remain unchanged.
The verification table uses two existing task identifiers and six additional fields:
| Field | Purpose |
|---|---|
paper_id, task_index | Link to the original benchmark task |
verification_status | Recorded task-level verdict |
verification_setup | Experimental conditions, including data or environment |
verification_result | Reported measurements or stage outcomes |
verification_scope | What the result supports under those conditions |
verification_source | Link to the recorded verification summary |
verification_git_diff | Unified diff for the corresponding experiment, when available |
The results documentation describes five quantitative examples covering forecasting, reasoning, multimodal retrieval, reinforcement learning and neural reconstruction. The measurements and their experimental conditions provide references for assessing research-task execution and the conclusions supported by the pipeline.
from datasets import load_dataset
results = load_dataset("Just-Curieous/EXP-Bench", "verification", split="audit")
The first 21 verification records refer to the May 20, 2026 progress snapshot. The additional 18439/task1 record reports the October 7, 2026 native-calibration trial. The original default configuration contains all 461 benchmark tasks.
verification_git_diff is reserved for the unified diff corresponding to the reported experiment. The 18439/task1 value contains the complete experimental-code patch and source/base/launch instructions; two machine-specific interpreter strings in logging metadata are replaced with python3 for publication. All 21 historical values remain empty because their corresponding experimental-code patches have not been recovered from the saved artifacts. Empty values do not mean that no code changes were made.
A usable patch should identify its source repository and base commit in comment headers, followed by the unified diff. Readers can check out that base commit, save the field as experiment.patch, and use git apply --check experiment.patch followed by git apply experiment.patch. Execution commands and data preparation must also be supplied before a complete reproduction is possible.