Agentic Determinism Index (ADI)

A public harness that asks one narrow question of hosted LLM APIs:

If I send you the exact same request N times, concurrently, and again across days, how identical are your answers?

No benchmark of intelligence, no quality judgment. Just reproducibility, measured at the byte level, per model, over time, from raw transcripts anyone can re-score.

• Zero dependencies. Python ≥ 3.9 standard library only. Clone and run.
• Methodology open, scores live. The harness and scoring spec (METHODOLOGY.md, v0.1 draft) ship with the first reference leaderboard. Challenge any metric via issues; scores remain recomputable from transcripts and update as new reference runs land.
• A door, not a wall. Alongside scores we document the exact conditions a provider would need to meet for reproducible serving. Providers that meet them get recognized for it.

Quickstart

git clone <this-repo> && cd agentic-determinism-index

# keys for whichever providers you want to probe (only the ones you set are usable;
# unset providers are skipped automatically)
export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
export GEMINI_API_KEY=...
export NVIDIA_API_KEY=...
export OPENROUTER_API_KEY=...
export HF_API_KEY=...

# edit configs/example.json to the models you care about, then:
python3 -m agentic_determinism_index run --config configs/example.json
python3 -m agentic_determinism_index score runs/reference/<timestamp>
python3 -m agentic_determinism_index site --run runs/reference/<timestamp> --out website/index.html

run fires each probe case at each target, a concurrent burst plus spaced serial requests, and writes raw transcripts (probes/*.json) plus an environment manifest. score turns a run directory into scores.json and a human-readable SCORES.md. site renders a standalone leaderboard page from scores.json into a static HTML file. Nothing is uploaded anywhere; everything stays on your disk unless you choose to contribute it.

What it measures

Per (provider, model, case):

Definitions and the probe protocol are specified in METHODOLOGY.md.

How this relates to the model or agent you are using

The scores here do not describe a model. They describe a serving tuple: (provider, model snapshot, parameters, serving stack) during a time window. The same weights served two different ways will score differently, determinism is a property of the deployment, not the weights.

That matters for you in three concrete ways:

1. Your agent inherits the reproducibility of the exact tuple it calls. If your agent calls some-model-2026-05-13 at temperature 0 with a tool schema, its reproducibility is that tuple's, not the model family's, and not what this repo measured for a different snapshot or parameter set. To measure your stack, copy your production request shape (same model string, temperature, seed, max tokens, JSON/tool schema) into a case file under cases/ and run the harness against it. The closer the probe is to your real traffic, the more the score means.

2. Probe through the same path your agent uses. If your requests go through a gateway, proxy, or router, that layer is part of your stack, point the harness at it, not at the provider directly. Any OpenAI-compatible endpoint works via the openai_compatible provider with a base_url, which also means you can probe a self-hosted vLLM/SGLang deployment and compare it against hosted APIs under identical cases.

3. Multi-step agents compound single-step variance. A single flipped token early in a plan can change every subsequent action. If a model's mode_share is 0.8 per call, a 10-call agent trajectory repeats far less often than 80% of the time. When your agent tests are flaky, this harness tells you how much of that is the serving layer before you debug your own code.

How scores are aggregated

Two strictly separated tiers:

• Reference runs, the only data that feeds the published leaderboard. Executed by the maintainers on a fixed cadence, with disclosed harness version, config, account tier, and region. Provider behavior varies by account tier, region, and routing, so leaderboard comparability requires a controlled, uniform vantage point.
• Community replications, contributed runs, published alongside the leaderboard as labeled replications that confirm or contradict the reference scores from other vantage points (different tiers, regions, days). They are never merged into reference scores. Contributing your data is a PR: see CONTRIBUTING.md.

Every published score, reference or community, is recomputable from its raw transcripts with python3 -m agentic_determinism_index score. A score you can't re-derive doesn't get published.

For public publication, run:

python3 -m agentic_determinism_index site --run-root runs/reference --out website/index.html

Then commit only website/index.html (or equivalent generated directory) alongside the corresponding reference run if you are maintaining a mirrored leaderboard. The generated page now includes an on-page Stack-drift timeline section that tracks (provider, model) stack-ID changes (system_fingerprint and modelVersion) across your scored reference run history.

Supported providers

Adding a provider is one adapter class in agentic_determinism_index/providers.py.

Status

v0.1, methodology comment window open, first reference scores published and updated continuously. License: MIT.

Maintained by

Lemma Ventures AG, Zug, Switzerland, which builds deterministic-inference infrastructure.

Incentive disclosure: Lemma has commercial interest in reproducible serving. This harness exists so published scores do not require trusting the maintainer: every score is recomputable from raw transcripts with python3 -m agentic_determinism_index score. Community replications are welcome; the reference leaderboard column is maintainer-controlled for tier/region comparability.

This repo measures serving reproducibility. It does not implement pinned-stack serving, receipts, or replay infrastructure.

Hosting the leaderboard site

The site command emits a single-file, self-contained website/index.html (no external assets, no tracking).

Live links (use these):

• Scores page: single-purpose leaderboard only. Footer names Lemma; no product CTAs.
• Do not embed the live leaderboard inside marketing pages. Other Lemma properties link out by URL.

Medals: each published snapshot ranks reference tuples; top 3 receive 🥇🥈🥉 (most reproducible under the disclosed protocol). Medals are snapshot-relative, not permanent certifications.

Bootstrap reference scores ship with the harness. A first analysis follows after about a month of continuous runs and open-source community contributions. Until then the page shows the latest bootstrap reference run and updates as new runs land.

To publish a snapshot:

python3 -m agentic_determinism_index site --run-root runs/reference --out website/index.html
git add website/index.html runs/reference/<stamp> && git commit -m "publish leaderboard for <stamp>"
git push origin main

GitHub Pages is served from /docs on main (GitHub only allows / or /docs for legacy Pages). website/index.html remains the build output; copy or regenerate into docs/ when publishing.

Background and motivation

Read the author's launch post for the full context and thoughts behind the Agentic Determinism Index:

https://lemma.ventures/blog/your-model-is-not-non-deterministic