Hopper 12B (trained): Decision Index run

A complete run of Decision Index edition 0.2.1 (150,317 scoreable requests) for Hopper 12B (trained). The model is a LoRA adapter, HopitAI/hopper on branch hopper-12b at revision 4b605f25eb903c2cf4d121040ffb097240b3bc37, on google/gemma-4-12B-it at revision 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7.

Scored with apolinario/decision-index at 87d4650: Decision Index 0.2.1 47.52 (raw 59.99). All 150,317 requests were answered (status ok).

How the run was made

• Where: run on partner hardware, except eight benchmarks we ran ourselves with the same package and adapter revision (Humicroedit, cfcolor, GPQA Diamond, ChessBench, Amazon ESCI, HLE, Habermas Machine, HoVer). Each row records its source (partner or hopit).
• Code: our public serving package, hopper at tag 12b-1.1.0 (package 1.2.0), hopper-serve --base-loader gemma-4-12b-it, with the readout map shipped in the package.
• Requests: one /v1/systemone request per question. Questions with more than 26 options use the package's disclosed tournament shortlist (k = 10).
• Adapter: served unmerged, as the package does.
• Inputs: the partner's rows come from a local rebuild of the 0.2.1 suite with the kit. With the kit's payload hash, 125,779 of those 126,316 rows are byte-identical to the official build; 537 differ in rendering (API-Bank 508, BFCL 10, When2Call 19). Re-running 450 of the API-Bank rows on the official inputs gave the same top answer for 95.1% of them and changed the Index by −0.05.
• Timings: the published rows carry no timestamps, and wall-time fields are set to 0.

We checked that every request ID of the edition is present exactly once and that every response is valid for its question before scoring. runs/hopper-12b-trained/environment.json lists the weights, code, suite and file hashes.

Versus the untrained base

Note (2026-10-05): We have not run the untrained base model (google/gemma-4-12B-it with the same serving package, prompt and readout) through the full suite. A 2,000-request sample we scored earlier projected it at about 48.7, with wide uncertainty (roughly 44–53). This trained adapter may therefore score below its own base under our serving. We will publish a full measurement if we run one.

These files are model outputs only and do not include the benchmark texts. The benchmarks keep their own upstream licences.