autotrust/GEV-26B-Decide

A decision model with adaptive thinking, on Gemma-4-26B-A4B-it (26 B parameters, ≈ 4 B active per token) — Decision Index 0.2.1: 62.48

Decision Index

How the score was computed (our scoring with the kit's score --edition 0.2.1, not a board entry):

• Knowledge & Reasoning: all ten benchmarks with adaptive thinking (per-benchmark table under Adaptive thinking).
• Other four areas: System 1 only (thinking off), from the complete System 1 run of these weights (all 150,759 requests, 0 errors), results in autotrust/jev-decision-index-results (runs/jev-gemma4-26b-a4b, the weights' previous name). Thinking was tried on five of their benchmarks and is not used there: it adds little to classification, retrieval and tool selection (see Outside Knowledge & Reasoning).
• The board requires a median latency of at most 1,000 ms per request, measured on an RTX PRO 6000. System 1 answers in about 45 ms on a B200; adaptive thinking is far slower on the Knowledge & Reasoning benchmarks (see Latency).

Details: reports/decision_index_adaptive.json, reports/adaptive_latency_summary.json.

New (3 October 2026): computer use and robot arm, 60–85 ms per step

The fastest vision model of the family. Every step below is one System 1 decision (thinking off): a screenshot or a camera image in, a probability for every action out, in a single forward pass on one B200.

Computer use: screenshot → which element to click. A real browser (headless Chromium). Every clickable element gets a numbered box; System 1 picks the next click (or "the task is complete"), the browser clicks it, and the loop repeats.

95% of 60 random multi-step tasks completed (shop, settings, mail; 3–7 clicks each) in about 85 ms per click: the same success rate as JEV-27B-VL, 3× faster. The colour swatches carry no text, so that click is decided from the screenshot alone.

Robot arm: pick and place from a camera image. At every step System 1 looks at the top camera image and answers two questions: is the target left or right of the gripper, and above or below it? The arm moves accordingly and halves its step whenever an answer flips. It grasps the cube, carries it and drops it in the tray (MuJoCo simulation).

61 ms per decision, so a whole pick and place takes 4–8 seconds of model time. It completed 40% of 20 random scenes: close to the target its left/right answers are less precise than JEV-27B-VL's, so more grasps miss. Once grasped, 8 of 9 cubes ended in the tray.

Same scenes and tasks for every model in the family:

• For computer use, give it the element text (as an accessibility tree would). With the numbered boxes alone it completes 15% and often declares the task complete too early.
• Use System 1 for simple visual questions inside a control loop. Asked to pick one of 8 motor commands directly, it completed 0 of 10 scenes.

Demo code: JEV-9B vl/demos/ (set JEV_URL to this server). Per-episode results: reports/demos/.

Overview

GEV-26B-Decide answers typed questions with a calibrated probability for every option; with thinking switched on, it thinks only when it needs to. System 1 decides in one forward pass (about 45 ms). When its leading option is uncertain, System 2 (the same backbone in Gemma-4 thinking mode) reasons over the question, and the reasoning is folded into the final probabilities. One set of weights, one vLLM engine, for text and images.

GEV-26B-Decide was previously published as autotrust/JEV-Gemma4-26B-A4B; the weights are the same.

Two models, two organisations. TypeSafe Jev 1.13 is the hosted, closed model made by TypeSafe AI. autotrust/GEV-26B-Decide is an independent open-weights model built by AutoTrust AI; it is not affiliated with, endorsed by, or a product of TypeSafe AI.

Watch it think

Each puzzle has one correct answer. System 1 answers in one pass; with thinking: "auto", System 2 thinks when System 1 is below 0.8 confidence, and its answer is folded into the probabilities. The videos show puzzles that System 1 got wrong; the tables below count all puzzles.

Minesweeper. Which hidden cell is certainly safe? System 1 is at chance (21.0 % against 25 %); with thinking, 86.0 %. These thoughts usually reach the 8,192-token budget, and the answer read at that point is still right most of the time.

Connect Four. Which column wins now, or stops the opponent from winning next move? 54.0 % → 99.5 %.

Wordle. Which word still fits all the colour feedback? 52.0 % → 100 %.

Sudoku. Which digit belongs in the highlighted cell? 76.0 % → 99.5 %.

One-move puzzles, 200 generated puzzles per game (threshold 0.8, budget 8,192 thinking tokens; text input, chess with the board image as well):

Whole games and perception, where thinking helps little or not at all:

Thinking pays off when the answer can be checked step by step against explicit rules: logic puzzles, tactics, constraints, arithmetic. It does not help perception (sketches), reflexes (Flappy Bird) or long-horizon play (2048, full Connect Four games), and every thought costs seconds. Details: reports/thinking_games.json.

Adaptive thinking

1. Fast distribution. System 1 returns p1 in one pass.
2. Think only when uncertain. If the leading option of p1 is below the threshold (default 0.8), System 2 reasons in Gemma-4's thinking mode over the same state, question and options. The reasoning length is yours to set, as with the base model: think_budget caps the thinking tokens (default: no cap beyond the context window; the evaluations below used 8,192).
3. Fold the reasoning in. When the thinking channel closes, the answer-letter distribution p2 is read in one step, and the result is p = ½ p1 + ½ p2. On its own, p2 is close to one-hot and over-confident; the equal mix keeps the reasoning's accuracy and System 1's calibration.

The threshold and the mix were chosen on 1,754 questions from six public sets that are not part of the Decision Index (test or validation splits, 300 random questions each; AQuA-RAT has 254):

Accuracy in %. Calibration is unchanged: ECE 0.035 for System 1 and 0.035 for adaptive. The adaptive mode reaches 96 % of the always-think gain while thinking on 48 % of the questions. Thinking length: median 2,282 tokens, 90th percentile 7,573; 91.6 % of the thoughts finish within the 8,192-token budget.

Decision Index, Knowledge & Reasoning

All ten benchmarks of the area (33,856 scored requests) ran through the server (POST /v1/decide, thinking: "auto", threshold 0.8, budget 8,192 thinking tokens) and were scored with the Decision Index kit:

Accuracy in %; "chance" is the kit's random baseline, and the Decision Index skill rescales accuracy so that chance is 0 (below chance counts as 0). The Knowledge & Reasoning area skill rises from 0.429 to 0.602.

• Real gains on science, code, causal and multi-step reasoning: GPQA Diamond, CRUXEval, CLadder, MMLU-Pro and BBH.
• HLE only returns to chance. System 1 is below chance (8.4 % against 16.4 %), and the questions where it is confident enough not to think are almost all wrong (1.6 %). Thinking brings HLE to 17.8 %, the level of guessing, so its skill stays near 0.
• Chess does not benefit. Two thirds of the ChessBench thoughts hit the 8,192-token budget; the answer read from a truncated thought is no better than System 1 (17.0 % against 17.7 % on those questions), while finished thoughts gain a little (19.8 % → 22.7 %). Answers read from truncated thoughts are also over-confident, so the equal mix does not damp them.

Outside Knowledge & Reasoning

Adaptive thinking with the same settings on five benchmarks of the other areas (run stopped after these):

Small gains on tool calls and intents, none on retrieval, and a loss on BANKING77, whose training split System 1 was trained on: the untrained System 2 often overrules a correct System 1 with an over-confident wrong answer. Use thinking for reasoning questions; keep it off for classification, retrieval and tool routing.

Latency

Thinking costs time. Over the area's ten benchmarks, 66 % of the requests thought at least once. Thinking length: median 8,192 tokens (the budget) on GPQA, HLE and ChessBench, about 5,400 on MMLU-Pro, 3,300 on BBH and 900 on GSM8K. Estimated single-request latency on one idle B200 (System 1 ≈ 45 ms, thinking ≈ 250 tokens per second): about 0.05 s without thinking and up to about 33 s with an 8,192-token thought; median 13.4 s over the area (90th percentile 33 s). With speculative decoding (below) thinking runs at about 438 tokens per second: median 7.7 s, 90th percentile 19 s. Lower threshold or think_budget to trade accuracy for speed; thinking: "off" keeps every decision in one pass.

MMLU-Pro and BBH (except six questions) ran with speculative decoding, which does not change the output distribution. Six BBH questions with 18 options hit a vLLM error in the speculative-decoding read-out path and were answered by the same server without speculative decoding; the current serve_decide.py reads answers through a path that works with speculative decoding.

Images

The checkpoint contains Gemma-4's vision encoder (no audio encoder), so System 1 and System 2 both accept images. The decision head was trained on text; decisions over images are zero-shot.

For reference, autotrust/JEV-27B-VL scores 78.3 % on VL-RewardBench with the same protocol.

Context length

The backbone's native context is 262,144 tokens (256K). We tested decisions that hinge on a single sentence placed at a random depth in long real text (concatenated PubMedQA abstracts): a yes/no question and a 16-option question, 10 of each per length, on one B200 with vLLM, one request at a time.

Lengths above 128K have not been tested yet.

Many options

choice takes 2–256 options. Up to 16 are read in one pass with the trained labels A–P. More options are read in groups of at most 16 (in parallel), then a final of 16; every option is read and none is pruned (strategy: "tournament", the default). Zero-shot intent classification, all options offered at once, 400 test utterances per row:

The 255-option sets merge three catalogues with overlapping intents, so part of the drop comes from near-duplicate labels. A single pass with labels beyond P (strategy: "single") is about 3× faster but less accurate here (CLINC150: 88.5 % against 95.2 %), so it is not the default. The BANKING77 and CLINC150 training splits are part of the training data (see below).

Quick start (vLLM)

hf download autotrust/GEV-26B-Decide --local-dir GEV-26B-Decide
bash GEV-26B-Decide/serve.sh          # vLLM on :8000; one GPU with 80 GB or more

serve.sh runs serve_decide.py: the standard vLLM OpenAI server (same flags as vllm serve) with a POST /v1/decide route. It loads the backbone once: plain requests are System 2, and requests for the LoRA module jev-decision (adapter_vllm/: backbone LoRA + the decision head as an lm_head LoRA) are System 1. It needs a vLLM build with Gemma-4 support plus patches/vllm-gemma4-lm-head-lora.patch (LoRA on Gemma-4's tied lm_head, vocabulary 262,144); tested with a vLLM development build from September 2026.

System 1 and adaptive thinking: POST /v1/decide

curl localhost:8000/v1/decide -H 'Content-Type: application/json' -d '{
  "kind": "choice",
  "state": "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball.",
  "question": "How much does the ball cost?",
  "options": ["$0.10", "$0.05", "$1.00", "$0.55"],
  "thinking": "auto"}'

The response has options, probabilities, choice, choice_index, usage and, when thinking was requested, thinking: {"used": true, "think_tokens": …, "think_seconds": …, "finished_within_budget": …}. GET /v1/decide/info lists the defaults.

import requests

def decide(kind, state, question, options=None, thinking="auto"):
    body = {"kind": kind, "state": state, "question": question, "thinking": thinking, **({"options": options} if options else {})}
    r = requests.post("http://localhost:8000/v1/decide", json=body).json()
    return dict(zip(r["options"], r["probabilities"])), r.get("thinking", {}).get("used")

decide("noul", "John was born on 29 February 1996.", "Was John's 7th birthday celebrated on a 29 February?")

System 2

requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "autotrust/GEV-26B-Decide",
    "messages": [{"role": "user", "content": "In one sentence, what is safety stock?"}],
    "max_tokens": 200, "chat_template_kwargs": {"enable_thinking": False}})

Speed (one B200, vLLM)

• System 1: median 45 ms for a single request; 257 decisions per second with 64 concurrent clients. vLLM matches the transformers engine below to a mean largest probability difference of 0.015 (300 held-out decisions).
• Adaptive thinking adds nothing when it does not think, and about 1 second per 250 thinking tokens when it does (see Latency).

Faster thinking with speculative decoding. MTP=1 bash GEV-26B-Decide/serve.sh adds Google's 0.9 GB draft model for this backbone (--speculative-config '{"model": "google/gemma-4-26B-A4B-it-assistant", "num_speculative_tokens": 4}'). On one B200 it speeds up System 2 about 1.8–1.9×: 247 → 438 tokens per second for a single request, 7,072 → 13,505 tokens per second at 128 concurrent requests (mean acceptance length 3.5–3.7 of 4). System 1 decisions are unchanged, but System 1 throughput at high concurrency drops (257 → 140 decisions per second with 64 clients); use it when you mostly think. serve_decide.py reads answers with the token restriction that works under speculative decoding, which needs --max-logprobs 256 (set in serve.sh).

If you call /v1/completions for System 1 yourself, pass top_k: 0 and top_p: 1.0: the model's generation config sets top_k=64 and top_p=0.95, which vLLM applies as request defaults and which would truncate the returned probabilities.

Usage (transformers + peft, System 1)

import json, torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from safetensors.torch import load_file
from transformers import AutoTokenizer, Gemma4ForConditionalGeneration

d = snapshot_download("autotrust/GEV-26B-Decide")
tok = AutoTokenizer.from_pretrained(d)
base = Gemma4ForConditionalGeneration.from_pretrained(d, dtype=torch.bfloat16, device_map="cuda")
# System 2: `base` is gemma-4-26B-A4B-it unchanged; use base.generate(...) (text or images).

# System 1: adapter merged in memory + head
m = PeftModel.from_pretrained(base, f"{d}/adapter").merge_and_unload().eval()
backbone = m.model
jc, T = json.load(open(f"{d}/judge_config.json")), json.load(open(f"{d}/calibration.json"))["per_kind"]
head = load_file(f"{d}/head.safetensors"); W, b = head["proj.weight"].cuda(), head["proj.bias"].cuda()

@torch.no_grad()
def decide(kind, state, question, options):
    lines = options if kind != "choice" else [f"{'ABCDEFGHIJKLMNOP'[i]}) {o}" for i, o in enumerate(options)]
    text = f"[kind] {kind}\n[state] {state}\n[question] {question}\n[options]\n" + "\n".join(lines) + "\n[decision]:"
    ids = torch.tensor([[tok.bos_token_id] + tok.encode(text, add_special_tokens=False)], device="cuda")
    h = backbone(input_ids=ids, use_cache=False).last_hidden_state[0, -1].float()
    z = 30.0 * torch.tanh((W @ h + b) / 30.0)
    s, _ = jc["slots"]["ranges"][kind]
    return dict(zip(options, torch.softmax(z[s:s + len(options)] / T[kind], 0).tolist()))

print(decide("noul", "Customer says the parcel arrived damaged and wants their money back.",
             "Is the customer asking for a refund?", ["false", "true"]))

This path reads up to 16 options per pass; for more, use the server (or read groups of 16 and a final, as above).

Engine and read-out

• Template bare-v1, prefixed with <bos>: [kind] … [state] … [question] … [options] A) … [decision]:
• Read-out: the final-norm hidden state of the last token goes through a linear fp32 head (hidden 2,816 → 24 slots), soft-capped at 30 like Gemma's own logits; inactive slots are masked, the logits are divided by the per-kind temperature and softmaxed. Nothing is generated.

Temperatures

Use calibration_gold.json when you gate automatic actions on confidence.

Training data (disclosure)

System 1 was trained on teacher distributions and ground-truth decision data. The ground-truth data includes the public training splits of some datasets whose test splits the Decision Index uses (among them BANKING77 and CLINC150); no test split of any benchmark was used, and suite items were excluded before training. The list has been provided to the Decision Index maintainers. MMMU / MMMU-Pro are not valid evaluations for this model. The adaptive-thinking settings were chosen on data outside the Decision Index suite.

Limitations

• Where the teacher is wrong, System 1 often is too, and it can be confidently wrong: on HLE the questions it answers without thinking are 1.6 % correct. Thinking does not help classification or retrieval and can hurt tasks System 1 was trained on (BANKING77 macro-F1 88.0 → 85.0). Adaptive thinking helps most on science, code, math and logic, does not help on chess or expert-level HLE questions, and can slightly lower accuracy on commonsense questions (CommonsenseQA 86.7 → 85.0).
• Thinking is slow on hard inputs: on expert-level and chess questions it often uses the full 8,192-token budget, and the adaptive mode does not meet the Decision Index latency limit on the Knowledge & Reasoning benchmarks.
• The Decision Index score above uses adaptive thinking on the Knowledge & Reasoning area and System 1 on the other four areas; it is our own scoring, not a board result.
• Weaker than JEV-27B on long structured inputs (e.g. the Decision Index's Home appliance simulator and POP909).
• HLE: below chance with System 1, like every open entry on the board.
• Decisions over images are zero-shot; contexts above 128K tokens are untested.
• The one-engine vLLM setup needs the bundled vLLM patch; noul and score accept only their canonical options.
• English-centric; not for high-stakes decisions without confidence gating.

Files

model-*.safetensors · config.json · processor_config.json · tokenizer* · chat_template.jinja · generation_config.json
                          google/gemma-4-26B-A4B-it, unchanged (System 2; text + image input)
adapter/                  System 1 LoRA (peft), for the transformers path
head.safetensors          24-slot decision head (fp32): proj.weight [24, 2816], proj.bias [24]
judge_config.json         slot layout, verbalizer ids, softcap, read-out
calibration.json          per-kind temperatures (default)
calibration_gold.json     per-kind temperatures calibrated against ground-truth answers
adapter_vllm/             System 1 for vLLM: backbone LoRA + the head as an lm_head LoRA, plus decision_head.json
serve_decide.py · serve.sh
                          vLLM server with POST /v1/decide (System 1, adaptive thinking) next to the OpenAI endpoints
patches/                  vLLM patch: LoRA on Gemma-4's tied lm_head
reports/                  adaptive thinking: validation summary, Decision Index recomputation, latency summary, other-area sample, games
reports/demos/            computer use and robot arm: per-episode results
videos/                   adaptive-thinking videos (Minesweeper, Connect Four, Wordle, Sudoku); computer use and robot arm

License

Apache-2.0 for the adapter, head and calibration files; base model under the Gemma 4 terms (https://ai.google.dev/gemma/docs/gemma_4_license). Not affiliated with TypeSafe AI.