ARC-Bench: An Open-Ended Autonomous-Research Benchmark Across Five Scientific Domains

The benchmark released with AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration.

---

ARC-Bench is a 55-topic, open-ended autonomous-research benchmark spanning five scientific domains. Each topic is not a fixed-input/fixed-output task — it is a research question plus a structured briefing. A research agent (or a human) must take a topic from question → experiment design → code → measurements → claims → write-up, and the deliverable is graded against a weighted, multi-criteria rubric.

The ML, quantum, and statistics topics are open research questions (the agent designs the experiment); the physics topics are published-paper reproductions (each scoped to a specific reference figure); the biology topics are constraint-based metabolic-modelling studies.

Load it

from datasets import load_dataset

# all 55 topics
ds = load_dataset("AIMING-Lab-UNC/ARC-Bench", split="test")

# a single domain subset
ml = load_dataset("AIMING-Lab-UNC/ARC-Bench", "ml", split="test")

row = ds[0]
print(row["id"], row["title"])
print(row["metric_key"], row["metric_direction"])

Available config names: default (all 55), ml, physics, quantum, biology, statistics.

Schema

Each row describes one benchmark topic. Deeply-nested / variable-shape fields are stored as JSON-encoded strings so the table schema is stable across all domains; parse them with json.loads.

Raw inputs

The flattened data/*.jsonl is convenient for load_dataset. The authoritative, human-readable benchmark inputs are also shipped verbatim under tasks/:

tasks/
├── meta_paper_quality.json         # shared paper-quality meta-rubric (manual grading)
└── <domain>/
    ├── topics.yaml                 # the domain topic registry
    ├── manifests/<ID>.yaml         # full per-topic briefing
    └── rubrics/<ID>.json           # weighted scoring rubric

How a topic is scored

Each topic carries a hierarchical rubric. For ML / quantum / statistics it has three buckets — Code Development, Code Execution, Result Analysis (weighted roughly 25 : 25 : 50). Physics and biology add a fourth Reproducibility bucket. Leaf criteria are graded on scientific substance and directional correctness of the evidence, not on rigid threshold matching (see each rubric's judging_note).

A second, optional layer — tasks/meta_paper_quality.json — grades the paper output (writing, code orchestration, figure quality, factual accuracy) and is intended for manual / vision-equipped grading rather than fast automated scoring.

Intended use

• Evaluating autonomous-research / AI-scientist agents end-to-end.
• Studying agent behavior across heterogeneous scientific domains with a single task format.
• As a stimulus set for human-in-the-loop or framework-comparison studies.

The runner harness, baseline adapters, and judges are not part of this dataset; they live in the source repository (link below).

Attribution

The benchmark glue (manifests, rubrics, registries) is the authors' own work. Some domain pipelines are driven by external Claude-Code agents, which should be credited when reporting domain results:

Links

• 💻 Code / harness: https://github.com/aiming-lab/AutoResearchClaw
• 📄 Paper (arXiv): https://arxiv.org/abs/2605.20025
• 🤗 Paper page: https://huggingface.co/papers/2605.20025

Citation

If you find ARC-Bench or AutoResearchClaw useful, please cite:

@misc{liu2026autoresearchclawselfreinforcingautonomousresearch,
      title={AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration},
      author={Jiaqi Liu and Shi Qiu and Mairui Li and Bingzhou Li and Haonian Ji and Siwei Han and Xinyu Ye and Peng Xia and Zihan Dong and Congyu Zhang and Letian Zhang and Guiming Chen and Haoqin Tu and Xinyu Yang and Lu Feng and Xujiang Zhao and Haifeng Chen and Jiawei Zhou and Xiao Wang and Weitong Zhang and Hongtu Zhu and Yun Li and Jieru Mei and Hongliang Fei and Jiaheng Zhang and Linjie Li and Linjun Zhang and Yuyin Zhou and Sheng Wang and Caiming Xiong and James Zou and Zeyu Zheng and Cihang Xie and Mingyu Ding and Huaxiu Yao},
      year={2026},
      eprint={2605.20025},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2605.20025},
}

License

Released under the MIT License.

_{Built with 🦞 by the AutoResearchClaw team}