BioMedArena

BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents

If you find BioMedArena useful, please give us a star on GitHub for the latest updates.

Reproducing and comparing deep research agents today is hard: the same backbone evaluated on the same benchmark can report different accuracies across papers because the harness and tool registry differ, and integrating a new model into a comparable evaluation surface costs weeks of model-specific engineering. These are symptoms of a broader reproducibility problem in deep research agent research. We release BioMedArena, an open-source toolkit that addresses this reproducibility gap and provides an arena for comparing biomedical deep research agents under a shared evaluation environment.

BioMedArena decouples six layers of biomedical agent evaluation: benchmark loading, tool exposure, tool selection, harness mode, context management, and scoring. The current public code exposes 166 registered benchmark entries (155 canonical benchmarks plus 11 deprecated compatibility aliases), 76 tools across 9 biomedical functional families, 4 modes, and 9 registered model backbone IDs. Adding a new model, benchmark, or tool reduces to registering a small provider adapter, loader, or schema/handler pair.

Overall Performance

Scroll horizontally to view all benchmark columns.

Documentation

The root README stays short on purpose. Detailed release information lives in docs/:

• Benchmark dataset inventory
• Tools and skills inventory
• API reference
• Metrics guide

Quick Check

After installing dependencies, run the offline smoke suite:

python3 scripts/run_quick_suite.py

Expected healthy output:

• 166 registered benchmarks
• 76 registered tools
• 4 registered modes
• 20/20 scorer checks passed

For the stricter offline release gate:

python3 scripts/release_gate.py --strict

Prepare the full BixBench agent setting only when you need the official capsule-backed protocol. This downloads the large CapsuleFolder-{uuid}.zip files explicitly instead of during normal benchmark loading:

biomedarena prepare-bixbench --revision main --extract
docker build -t biomedarena/bixbench-sandbox:latest docker/bixbench
biomedarena run --benchmark bixbench --bixbench-form open --bixbench-capsules \
  --backbone gemini-3-flash-preview --tools biomed --reasoning-mode heavy

If you are running fully offline from a cache you prepared yourself, add --bixbench-offline-metadata to the run command. The open-form path is official-compatible: it uses the public BixBench rows, mounted data capsules, and eval_mode-specific scoring, while the external FutureHouse evaluator is not vendored in this repository.

Installation

git clone https://github.com/AI-in-Health/BioMedArena.git
cd BioMedArena

python3.11 -m venv .venv
source .venv/bin/activate

python -m pip install -e ".[dev,eval,provider-gemini]"

cp .env.example .env

Fill at least one model provider key in .env:

OPENAI_API_KEY=<your-openai-api-key>
ANTHROPIC_API_KEY=<your-anthropic-api-key>
GEMINI_API_KEY=<your-gemini-api-key>
HF_TOKEN=<your-huggingface-token-for-gated-benchmarks>

Gated HuggingFace datasets also require accepting the dataset terms in the browser before HF_TOKEN can load them. See .env.example for optional domain-specific keys such as NCBI, OMIM, Serper, and Jina.

Basic Usage

List available resources:

biomedarena list-benchmarks
biomedarena list-backbones
biomedarena list-modes

The package name and command-line entry point are both biomedarena. Environment variables use the BIOMEDARENA_ prefix.

Run one benchmark cell:

biomedarena run \
  --benchmark medcalc \
  --backbone gemini-2.5-flash \
  --tools biomed --reasoning-mode light \
  --limit 5 \
  --output result.json

Run a small matrix cell:

python3 scripts/run_matrix.py \
  --config configs/matrix_default.yaml \
  --only medcalc,gemini,simple_llm \
  --limit-override 1

For the 7-setting quick-start experiment suite used to compare thinking, domain tools, web search, and combined tool use, see quick_run.sh.

Check official source accessibility before spending model budget:

python3 scripts/verify_benchmark_sources.py --benchmarks all

Execution Modes

The public CLI exposes four modes:

A unified CLI interface is also available via --tools / --reasoning-mode / --enable-thinking flags, which map to the modes above:

The legacy --mode / --web-tools flags remain supported for backward compatibility. Add --self-consistency to wrap any mode with majority voting.

Security

python_exec can execute model-supplied Python with timeout and basic denylist checks. Treat this as a convenience guard, not a hardened sandbox. Run untrusted workloads in an isolated container or VM, keep secrets out of the working directory, and disable code-execution or web-search tools for private data unless you have reviewed the policy.

External tools may call third-party APIs and public databases. Review the benchmark and tool inventories before running sensitive workloads.

Testing

python3 scripts/run_quick_suite.py
python3 scripts/release_gate.py --strict
python3 -m pytest tests/unit -q
HF_HOME=/tmp/biomedarena_hf_empty \
HF_DATASETS_CACHE=/tmp/biomedarena_hf_datasets_empty \
HF_TOKEN= HUGGING_FACE_HUB_TOKEN= HUGGINGFACE_HUB_TOKEN= \
python3 -m pytest tests/smoke -q -m "not slow"

Citation

@article{wu2026biomedarena,
  title={BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents},
  author={Wu, J and Zhou, H and Zeng, M and Zhu, J and Wu, J and Pan, J and Noori, A and Wu, S and Wu, H and Liu, F and Clifton, D A},
  journal={arXiv preprint arXiv:2605.06177},
  year={2026}
}

License

See LICENSE. Ported life-science skill attribution is tracked in harness/tools/openai_ported/NOTICE.md.