Open-source benchmark for evaluating AI code review tools — the datasets, the judge, and the pipeline code. Reproduce our results or evaluate your own tool.

The problem

As AI agents write more code, we need systems to make sure the code they generate is good. This has led to the proliferation of AI code review tools.

Without shared evals for these tools, every company grades its own homework. You can't reproduce the results, compare tools on the same dataset, or verify the methodology. With static evals, agents can game the benchmark. By creating an online and offline benchmark that can check each other, this project allows for robust code review evals.

We open-source everything: the PRs, the golden comments, the LLM judge prompts, the evaluation pipeline, and a continuously-updated online benchmark that avoids training data leakage.

Two benchmarks

Offline — fixed dataset, reproducible results

50 PRs from 5 major open-source projects, each with human-verified golden comments — the real issues a reviewer should catch.

Each PR has curated golden comments (173 total) with severity labels (Low / Medium / High / Critical) and category tags (bug, security, concurrency, data, api, perf, test_gap, doc_defect, style, speculative). An LLM judge matches each tool's review against the golden comments using three judge models (Claude Opus 4.5, GPT-5.2, Claude Sonnet 4.5). Category-based scoring profiles (Strict / Core / All) control which issue types count toward the score, and F-beta weighting lets users prioritize recall over precision.

Tools evaluated: Augment, Baz, Claude Code, CodeAnt, CodeRabbit, Cubic, Cursor Bugbot, Devin, Gemini, GitHub Copilot, GitLab Duo, Graphite, Greptile, KG, Kodus, Macroscope, Qodo, Sourcery, and more. Running a tool that isn't on the leaderboard takes an afternoon — fork the benchmark PRs, trigger the tool, run the pipeline. Publishing it alongside the others additionally requires meeting the inclusion criteria.

Known limitation: Static datasets risk training data leakage — tools may have seen these PRs during training. That's why we also run the online benchmark.

See offline/README.md for setup and usage.

Online — continuous, fresh PRs, no data leakage

The online benchmark continuously samples fresh real-world PRs from GitHub where code review bots left comments. Because the PRs are recent, tools can't have memorized them during training.

GitHub Search API
        │
        ▼
    ┌────────┐     ┌─────────┐     ┌─────────┐     ┌────┐     ┌───────────┐
    │Discover│────▶│ Enrich  │────▶│ Analyze │────▶│ DB │────▶│ Dashboard │
    └────────┘     └─────────┘     └─────────┘     └────┘     └───────────┘
   Search API      GitHub API     LLM 3-step      Postgres    Interactive
   finds merged    fetches full   extraction &    or SQLite   filters &
   bot-reviewed    PR context     matching                    time series
   PRs

How analysis works:

1. Extract bot suggestions — The LLM reads the diff the bot reviewed and the bot's comments, then extracts each actionable suggestion with its category (bug, security, performance, style, ...) and severity.
2. Extract human actions — The LLM reads the post-review commits and identifies what the developer actually fixed after the bot commented.
3. Judge matching — The LLM determines which bot suggestions correspond to actual fixes, producing per-PR precision (what % of the bot's comments were useful?) and recall (what % of real issues did the bot catch?).

Bots tracked: CodeRabbit, GitHub Copilot, Claude, Cursor, Augment, Codex, Gemini, Greptile, Graphite, Qodo, Propel, and others.

Dashboard features: Filter by language, project domain, PR type, issue severity, diff size, engagement signals (human comments/commits after bot review), solo-bot PRs, and sample controls. Track performance over time. Adjustable F-beta weighting. See online/FILTERS.md for the full filter spec.

See online/README.md for architecture and setup.

How the LLM judge works

Both benchmarks use an LLM-as-judge approach, but with different methodologies suited to their data:

In both cases, the judge prompt asks "do these describe the same underlying issue?" — different wording is fine, only the substance matters.

Judge model variance: Different LLM judges can score differently. We mitigate this by storing results per judge model and reporting which model was used. The offline benchmark has been evaluated with Claude Opus 4.5, Claude Sonnet 4.5, and GPT-5.2 — the top 5 tools are identical across all three judges, with most tools varying by at most 2 rank positions.

Repository structure

├── offline/                       # Offline benchmark (fixed dataset)
│   ├── golden_comments/           #   Human-curated issues per repo (5 JSON files)
│   ├── code_review_benchmark/     #   Pipeline: fork, download, extract, dedup, judge, export
│   ├── analysis/                  #   Interactive HTML dashboard
│   ├── tests/                     #   Test suite (no network access required)
│   └── results/                   #   Evaluation outputs (per judge model)
│
├── online/                        # Online benchmark (continuous)
│   ├── etl/                       #   Python pipeline
│   │   ├── pipeline/              #     Discover → Enrich → Assemble → Analyze → Label
│   │   ├── llm/                   #     Prompts, schemas, async client
│   │   ├── db/                    #     Database layer (SQLite + PostgreSQL)
│   │   ├── jobs/                  #     Background workers
│   │   └── dashboard/             #     Streamlit dashboard
│   └── api_service/               #   Rust API + embedded HTML dashboard
│
└── LICENSE                        # MIT

Quick start

Offline benchmark

cd offline
uv sync
cp .env.example .env               # add GitHub token + LLM API key

# Download reviews for all tools
uv run python -m code_review_benchmark.step1_download_prs --output results/benchmark_data.json

# Extract individual issues from reviews
uv run python -m code_review_benchmark.step2_extract_comments

# Deduplicate candidates (prevents false positives from inline+summary overlap)
uv run python -m code_review_benchmark.step2_5_dedup_candidates

# Run the LLM judge (pass dedup groups to avoid penalising duplicate candidates)
uv run python -m code_review_benchmark.step3_judge_comments \
  --dedup-groups results/${MARTIAN_MODEL}/dedup_groups.json

# View results
open analysis/benchmark_dashboard.html

Online benchmark

cd online/etl
uv sync
cp .env.example .env               # add GitHub token + GCP project + LLM API key

# Discover recent PRs from BigQuery
uv run python main.py discover --all --days-back 7

# Enrich with GitHub API data
uv run python main.py enrich --chatbot "coderabbitai[bot]" --one-shot

# Run LLM analysis
uv run python main.py analyze --all

# Launch dashboard
uv run python main.py dashboard

Adding a new tool

Running a tool that isn't on the leaderboard is open to anyone:

1. Fork the 50 benchmark PRs into a GitHub org where your tool is installed
2. Let the tool review each PR
3. Add the tool name to the download config and run the pipeline
4. Compare the results against the existing tools in the dashboard

See offline/README.md for detailed instructions.

Inclusion criteria

Publishing a tool on the leaderboard alongside the others has two further requirements.

Public usage. Roughly at least 600–1,000 reviewed public PRs, across a spread of orgs, repos, and authors. The offline benchmark is a fixed set of 50 PRs, so on its own it can't tell us whether a score reflects how a tool behaves in practice. We validate it against the online benchmark, which measures how developers respond to a tool's reviews in the wild, and that cross-check needs enough public review activity to be meaningful. Private installs aren't visible to us and can't be counted, so this is a measurement constraint rather than a judgement about a tool's overall adoption.

Attributable reviews. We need to be able to tell from the GitHub API that a review came from the tool rather than from a person — a bot account, a dedicated machine account, or a consistent marker in the comment body all work. Without one of those we can't separate a tool's findings from a human reviewer's comments.

For any tool we publish we also run the pipeline ourselves rather than take submitted results, so every number on the leaderboard is produced the same way.

If a tool doesn't meet these yet, the harness is public and you're welcome to run it and publish your own results.

Contributing

We welcome contributions — new tools, better golden comments, improved judge prompts, additional datasets. Open an issue or PR.

Citation

If you use this benchmark in your research or product evaluation, please cite:

@misc{code_review_benchmark,
  title   = {Code Review Bench},
  author  = {Aleksandr Zverianskii and Ashley Zhang and Jacob Clyne and Antía Garcia and Fazl Barez and Shriyash Upadhyay},
  url     = {https://github.com/withmartian/code-review-benchmark},
  year    = {2026},
  license = {MIT}
}

License

MIT — see LICENSE.