SciAgentArena

News! We are collaborating with several LLM providers (Qwen, Seed, Gemini, HY, etc.) to evaluate SciAgentArena. We are pleased to see that our benchmark is contributing to the advance!

A benchmark platform for evaluating AI agents across diverse scientific and biomedical domains. This repository provides agent implementations, standardized evaluation benchmarks, and a web-based submission interface.

Overview

SciAgentArena enables reproducible evaluation of AI agents on real-world scientific tasks spanning single-cell genomics, spatial transcriptomics, drug discovery, electronic health records, clinical genetics, and more. Each agent is evaluated against structured benchmark tasks with automated scoring.

Repository Structure

SciAgentArena/
├── agentdir/          # Agent implementations and setup guides (16 agents)
├── evaluations/       # Benchmark datasets and evaluation notebooks
│   ├── sc/            # Single-cell RNA-seq
│   ├── sp/            # Spatial transcriptomics
│   ├── ehr/           # Electronic health records
│   ├── genetics/      # Statistical genetics
│   └── cross_domain/  # eQTL and multi-omics
└── front_web/         # Web-based evaluation platform (Node.js + Python)

Benchmark Domains

The platform covers 7 benchmark families with 198 evaluation tasks:

Agents

Biomedicine & Omics

Chemistry & Drug Discovery

Clinical & Genetics

General Purpose

Web Evaluation Platform

The front_web/ directory contains a production-oriented evaluation platform:

• Submission modes: Submit Python pipeline code or a pre-computed .h5ad file
• Output: Per-task scores, aggregate metrics, execution logs, and overall average score
• Upload limit: Up to 2 GB
• Execution timeout: 15 minutes per submission

Starting the Platform

cd front_web
npm start

Then open:

• Home: http://localhost:3000
• Single-cell benchmark: http://localhost:3000/benchmarks/single-cell
• Spatial benchmark: http://localhost:3000/benchmarks/spatial

API Endpoints

Runtime Requirements

Node.js (web layer) + Python 3.10+ (evaluation):

pip install anndata scanpy numpy pandas scikit-learn
# Optional (for advanced metrics):
pip install scib scib_metrics squidpy scipy

Submission Contract

Code mode — your Python script is executed with these variables preloaded:

• path / dataset_path / DATASET_PATH: path to the selected .h5ad dataset
• np, pd, sc: numpy, pandas, scanpy

The script must assign its final output to an adata variable of type AnnData.

h5ad mode — upload a pre-generated .h5ad file containing the expected fields and embeddings.

ToolUniverse Agent Quick Start

The ToolUniverse agent (agentdir/ToolUniverse/) is the reference agent template for the platform.

Install

cd agentdir/ToolUniverse
pip install -r requirements.txt
cp .env.example .env   # fill in OpenAI or Azure OpenAI credentials

Adding a New Agent

1. Create a directory under agentdir/<agent-name>/.
2. Add a markdown file (e.g., README.md or <AgentName>.md) describing:
   • Category, repository link, and overview
   • System requirements and installation steps
   • How to run the agent on a benchmark prompt
   • Expected input/output format
3. Ensure outputs are compatible with the platform's submission contract (an AnnData assigned to adata, or a .h5ad file).

Adding a New Benchmark

1. Add a new benchmark entry in front_web/judge.config.json.
2. Add dataset presets pointing to the benchmark id.
3. Add evaluator logic in front_web/templates/pipeline_evaluator.py.
4. Add page copy, starter code, and override fields in front_web/public/benchmark.js.
5. Visit /benchmarks/<your-benchmark-id>.

Acknowledgements:

• ToolUniverse: https://zitniklab.hms.harvard.edu/ToolUniverse/
• AutoBA: https://github.com/JoshuaChou2018/AutoBA
• CellForge: https://github.com/gersteinlab/CellForge
• STELLA: https://github.com/zaixizhang/STELLA
• ChemCrow: https://github.com/ur-whitelab/chemcrow-public
• CACTUS: https://github.com/pnnl/cactus
• ChemToolAgent: https://github.com/OSU-NLP-Group/ChemToolAgent
• DELTA: https://github.com/deltawave-tech/delta
• LIDDiA: https://github.com/ninglab/LIDDiA
• DrugAgent: https://github.com/FermiQ/drugagent
• Medea: https://github.com/mims-harvard/MEDEA
• TxAgent: https://github.com/mims-harvard/TxAgent
• MRAgent: https://github.com/xuwei1997/MRAgent