Every model on the same 22,773 test records. Down and to the right is better; the stars are open models fine-tuned on their own decision readout with a coherence penalty.

Contents — Quick start · The data · Why coherence · Leaderboard · Highlights · Published models · Jev 1.13.0 baseline · Behavioral probes · Vision · Latency · What is in this repo · Label audit and changelog

Quick start

A System One model (TypeSafe's Jev, or any open model Jevified by the engine) does not write text. It reads a state, answers typed questions, and returns probability distributions your code can branch on. The product claim is calibration: an answer given 0.8 should be right about 80% of the time. Most benchmarks can only check the argmax; jev-bench checks the distribution — four configs carry the human vote shares behind each label (go_emotions, measuring_hate_speech, civil_comments, chaosnli).

Every row is one (state, question, label) triple in exactly the wire format a System One model consumes — send state and question to POST /v1/systemone as-is.

from datasets import load_dataset
import json

ds = load_dataset("Praveenrajus/jev-bench", "chaosnli", split="test")
row = ds[0]
state, question = json.loads(row["state"]), json.loads(row["question"])
label, human = row["label"], json.loads(row["soft_label"])   # human = {"entailment": 0.63, "neutral": 0.37, ...}

state, question and soft_label are JSON strings so every config shares one stable schema; label is a string (option key for choice, level index for score, "0"/"1" for noul). The jevify package gives you BenchRecord.from_row, the metrics (ECE, Brier, RPS, selective accuracy, TVD to human), the API runner and the figures.

The data

Natural label distributions everywhere (uniform random samples, seed 20260920); a calibration benchmark must not shift base rates. Splits: test ≤ 1,000 per config (2,000 for civil_comments, all of ChaosNLI), validation ≤ 500, train ≤ 8,000 so Tier 1/2 recipes and temperature scaling have in-distribution data without touching test.

Choice

Score

Noul

Licenses are those of the upstream datasets; this repackaging adds no restrictions. Per-source provenance is in manifest.json.

Why we measure coherence

A decision model answers many questions about the same thing, and its answers should fit together. If a model gives a support message 60% for "billing" in a three-way routing question but 45% when asked "is it billing?", it has contradicted itself: no single belief about the world produces both answers. Such answers are incoherent, and there is a classical result (de Finetti) that incoherent probabilities are never better: a coherent set of answers always exists that scores at least as well whichever answer turns out to be right. Accuracy and calibration cannot see this: a model can be accurate and well calibrated on each question alone and still contradict itself across related ones.

We measure it with the sure loss d². For each family of related questions (a choice question plus a yes/no question for each option; a yes/no question plus its negation; a rating plus "at least level k" thresholds; 4,749 families built automatically from the test items) we find the nearest set of answers that could all be true together and take the squared distance. Zero is perfectly coherent. For scale: Jev 0.081; untuned open models 0.14 to 0.43; plain supervised readout fine-tuning 0.27 to 0.37; with the coherence penalty 0.02 to 0.04; Perplexity's hosted decider 0.219.

It matters because software that acts on a probability often asks several related questions about one input (route it? is it billing? is it urgent?). If the answers disagree, no single number can be trusted. Coherence is what lets you ask once and rely on the answer. Limits: it is measured only on the relations our generator builds, and a model can be perfectly coherent and wrong. Details and the training penalty: FINDINGS 18.

Leaderboard

Jev and the best of every tier (all 46 rows below). Tier 0 is zero training: prompt, logit readout, and a recipe fitted on validation splits only. Tier 1 adds decision heads trained with proper scoring rules; Tier 2 adds LoRA under those heads. Readout FT trains the model on its own decision distribution, optionally with a coherence penalty (FINDINGS 18). External is another team's model. Six sources are held out of every training run.

All 46 rows, and accuracy / ECE per config for every drawn model

model	tier	macro acc	macro ECE	macro Brier	sel@90	choice acc	score acc	noul acc	TVD→human	GPU	test cost
Jev 1.13.0 (TypeSafe API)	API	0.733	0.113	0.349	0.760	0.770	0.503	0.881	0.432
Qwen/Qwen-AgentWorld-35B-A3B (readout LoRA + coherence)	Readout FT	0.770	0.055	0.292	0.799	0.792	0.566	0.917	0.305
Qwen/Qwen3.5-9B (readout LoRA + coherence)	Readout FT	0.763	0.056	0.299	0.791	0.791	0.554	0.906	0.301
Qwen/Qwen3.5-9B (readout LoRA)	Readout FT	0.759	0.052	0.302	0.787	0.790	0.541	0.905	0.314
Qwen/Qwen3.5-4B (readout LoRA + coherence)	Readout FT	0.751	0.058	0.316	0.779	0.771	0.555	0.893	0.303
perplexity-ai/pplx-decider-v1-27b (hosted API, as served)	External	0.748	0.080	0.318	0.774	0.775	0.519	0.911	0.378		$0.33
Qwen/Qwen3.5-4B-Base (readout LoRA + coherence)	Readout FT	0.747	0.058	0.318	0.777	0.771	0.535	0.897	0.314
Qwen/Qwen3.5-4B (Tier 2 residual)	Tier 2 residual	0.747	0.110	0.342	0.774	0.770	0.529	0.903	0.347	A40
Qwen/Qwen3.5-4B (readout LoRA)	Readout FT	0.743	0.059	0.321	0.772	0.766	0.530	0.896	0.326
google/gemma-4-E4B-it (readout LoRA + coherence)	Readout FT	0.742	0.053	0.322	0.772	0.759	0.546	0.889	0.297
Qwen/Qwen3.5-4B-Base (readout LoRA)	Readout FT	0.741	0.064	0.324	0.770	0.768	0.522	0.893	0.324
google/gemma-4-E4B-it (readout LoRA)	Readout FT	0.737	0.057	0.330	0.766	0.754	0.531	0.892	0.316
Qwen/Qwen3.5-4B (Tier 2 residual, lr 3e-05)	Tier 2 residual	0.734	0.096	0.342	0.762	0.761	0.518	0.884	0.337	A40
google/gemma-4-12B-it	Tier 0	0.716	0.086	0.359	0.740	0.749	0.507	0.855	0.405	A40
Qwen/Qwen3.5-9B (Tier 1 residual, 6-epoch cap, mean of 3 seeds)	Tier 1 residual	0.714	0.071	0.362	0.740	0.725	0.507	0.878	0.376	A40
Qwen/Qwen-AgentWorld-35B-A3B	Tier 0	0.709	0.077	0.350	0.735	0.754	0.473	0.852	0.408	L40S
google/gemma-4-E4B (readout LoRA)	Readout FT	0.708	0.061	0.359	0.736	0.717	0.502	0.874	0.335
Qwen/Qwen3.5-2B (readout full fine-tune + coherence)	Readout FT	0.704	0.056	0.358	0.735	0.718	0.510	0.853	0.307
togethercomputer/Tev1-4B-experimental (Tev1 prompt)	External	0.703	0.086	0.361	0.732	0.717	0.512	0.850	0.394	A40
Qwen/Qwen3.5-2B (readout full fine-tune)	Readout FT	0.703	0.056	0.358	0.734	0.715	0.509	0.855	0.315
Qwen/Qwen3.5-2B-Base (readout LoRA + coherence)	Readout FT	0.702	0.051	0.365	0.731	0.714	0.509	0.852	0.320
Qwen/Qwen3.5-2B (readout LoRA + coherence)	Readout FT	0.702	0.054	0.364	0.729	0.718	0.496	0.858	0.315
Qwen/Qwen3.5-2B (readout LoRA)	Readout FT	0.701	0.051	0.362	0.730	0.713	0.508	0.849	0.324
Qwen/Qwen3.5-4B (Tier 1 residual)	Tier 1 residual	0.698	0.089	0.378	0.729	0.699	0.507	0.862	0.360	A100-80GB	$1.28
Qwen/Qwen3.5-2B-Base (readout LoRA)	Readout FT	0.696	0.053	0.365	0.725	0.714	0.487	0.852	0.342
Qwen/Qwen3.5-2B (Tier 2 residual, lr 3e-05, soft labels)	Tier 2 residual	0.694	0.105	0.390	0.724	0.697	0.497	0.859	0.364	A40
togethercomputer/Tev1-4B-experimental (Jevify prompt)	External	0.690	0.088	0.369	0.718	0.714	0.487	0.834	0.408	A40
Qwen/Qwen3.5-2B (Tier 2 residual, lr 3e-05)	Tier 2 residual	0.690	0.103	0.389	0.720	0.693	0.510	0.840	0.332	A40
Qwen/Qwen3.5-9B	Tier 0	0.689	0.092	0.379	0.717	0.716	0.503	0.815	0.423	A40
Qwen/Qwen3.5-2B (Tier 2 residual)	Tier 2 residual	0.685	0.116	0.410	0.713	0.671	0.508	0.854	0.370	A40
Qwen/Qwen3.5-2B (Tier 2 residual, soft labels)	Tier 2 residual	0.674	0.116	0.412	0.703	0.682	0.471	0.837	0.433	A40
Qwen/Qwen3.5-4B (Tev1 prompt)	Tier 0	0.673	0.104	0.402	0.702	0.687	0.499	0.805	0.421	A40
Qwen/Qwen3.5-4B	Tier 0	0.662	0.090	0.401	0.689	0.687	0.468	0.796	0.438	A100-80GB	$0.96
google/gemma-4-E4B-it	Tier 0	0.658	0.092	0.398	0.682	0.700	0.432	0.798	0.434	A100-80GB	$1.00
Qwen/Qwen3.5-2B (Tier 1 residual)	Tier 1 residual	0.632	0.069	0.445	0.657	0.596	0.449	0.835	0.374	A100-80GB	$0.67
Qwen/Qwen3.5-2B (Tier 1 residual, 6-epoch cap, mean of 5 seeds)	Tier 1 residual	0.631	0.078	0.450	0.655	0.603	0.445	0.827	0.373	A40
IFM/K2-Horizon-7B	Tier 0	0.623	0.093	0.436	0.647	0.617	0.409	0.813	0.434	A40
Qwen/Qwen3.5-2B (Tier 1 replace)	Tier 1 replace	0.599	0.083	0.475	0.621	0.567	0.378	0.828	0.416	A100-80GB	$0.62
google/gemma-4-E2B-it	Tier 0	0.591	0.123	0.484	0.612	0.594	0.440	0.716	0.492	L4	$0.75
Qwen/Qwen3.5-2B	Tier 0	0.577	0.089	0.485	0.599	0.545	0.424	0.750	0.458	A100-80GB	$0.66
HuggingFaceTB/SmolLM3-3B	Tier 0	0.552	0.111	0.519	0.570	0.504	0.401	0.744	0.460	A100-80GB	$0.64
Qwen/Qwen3.5-0.8B	Tier 0	0.526	0.112	0.542	0.544	0.436	0.388	0.759	0.440	L4	$0.55
Qwen/Qwen3.5-0.8B-Base	Tier 0	0.466	0.105	0.582	0.478	0.326	0.412	0.692	0.452	L4	$0.50
IFM/K2-Horizon-0.9B	Tier 0	0.448	0.119	0.589	0.461	0.344	0.343	0.670	0.479	L4	$0.35
google/gemma-4-E2B	Tier 0	0.396	0.115	0.609	0.404	0.247	0.382	0.600	0.496	L4	$0.69
CLM-v0.1-8B (Contrastive-LM, self-hosted)	External	0.340	0.338	0.814	0.349	0.213	0.234	0.593	0.669

Highlights

Jev, measured

• Excellent and calibrated on crisp, grounded questions (ARC 0.979 at ECE 0.010, FEVER 0.972) — and over-confident exactly where humans disagree: on ChaosNLI its confidence stays at 0.81–0.88 while its accuracy runs 0.47 → 0.96 across human-agreement bands (r = 0.046). Baseline
• Applies a rule written into the question at 0.924 — its largest lead over every open model. New tests
• ~75 ms + 5.5 µs per input token, flat in the number of options: the answer is read, not decoded. Latency

Open models, no training (Tier 0)

• A readout plus a validation-fitted temperature matches Jev's calibration, not its accuracy: Gemma-4-12B-it 0.716 / ECE 0.086, Qwen3.5-9B 0.689 / 0.092.
• On a phishing benchmark written by someone else, untrained Gemma-4-12B beats Jev by twenty points (0.825 vs 0.628). Community benchmarks

Training on the decision itself (post-training study, FINDINGS 18)

• Readout fine-tuning puts open models above Jev overall: 9B 0.763 / ECE 0.056, 4B 0.751 / 0.058, Gemma-4-E4B 0.742 / 0.053, with TVD to human labels ~0.30 against Jev's 0.43.
• Trained alone, each question gets sharper while its implied siblings drift: answers to related questions contradict each other more (sure loss 0.15 → 0.28 at 4B). A coherence penalty cuts that tenfold, to 0.02–0.03 — below Jev's 0.081 — at no accuracy cost, at every size, in both families and under full fine-tuning.
• Starting from the base checkpoint ends in the same place in Qwen3.5 (4B: 0.741 vs 0.743) and closes most of Gemma's wider gap.
• A 35B mixture-of-experts model has the highest accuracy in the table. Qwen-AgentWorld-35B-A3B (about 3B active parameters), readout fine-tuned with the coherence penalty: 0.770 accuracy, ECE 0.055, TVD 0.305, sure loss 0.030 (one seed), up from 0.709 / 0.077 / 0.408 / 0.138 untuned. It gives back accuracy on two rating scales (yelp5 62.3 to 52.2, sst5 56.7 to 52.2). Model.

What post-training does to a decision readout (FINDINGS 17)

• Base checkpoints range from under- to over-confident (fitted temperature 0.6–1.7). Where the SFT checkpoint is published, SFT leaves the readout close to calibrated, and preference optimisation (Tülu DPO, SmolLM3 APO) then makes it over-confident at flat accuracy — two labs, two algorithms.
• The first post-training step makes decisions more invariant in all eight base → post-trained pairs; Gemma-4's instruct models score below their bases without their chat template.

Beyond jev-bench

• Fine-tunes stop flagging phishing while keeping their ranking — a moved threshold, repaired by one shift fitted on sixteen labelled emails. Phishing threshold
• Readout fine-tuning cuts injected-instruction hijacks from ~0.3 to ~0.1. New tests

Results log — every result as it was found (later entries supersede earlier ones)

• Post-training shapes the readout stage by stage (results/post-training/; FINDINGS 17). Five families scored at every published stage with one battery: base checkpoints range from under- to over-confident (fitted temperature 0.6–1.7), SFT leaves the readout close to calibrated where its checkpoint is published, and preference optimisation (Tülu DPO, SmolLM3 APO — two labs, two algorithms) then makes it over-confident at flat accuracy (temperatures up 1.3–2.4×). The first post-training step makes decisions more invariant in all eight base → post-trained pairs; Gemma-4's instruct models score below their bases without their chat template.
• Readout fine-tuning with a coherence penalty gives the best open decision models here (FINDINGS 18). Trained on its own decision distribution, Qwen3.5-4B reaches 0.751 macro accuracy at ECE 0.058 and TVD 0.303 to human labels (Jev 0.733 / 0.113 / 0.432); without the penalty, fine-tuning makes answers to related questions contradict each other (sure loss 0.15 → 0.28), and the penalty cuts that tenfold, to 0.029 — below Jev's 0.081 — at no accuracy cost. Same at 2B, 9B and on Gemma-4-E4B; starting from the base checkpoint ends in the same place. Models: jevify-qwen3.5-4b-readout-coh and nine siblings (2B/4B/9B, Gemma-4-E4B, base starts), each with a reproduction check.
• Given a rule, Jev applies it; no open model does yet. Three new tests (results/b/): 885 human-written LegalBench fact patterns with the rule written into the question, "none of the above" with the right option removed, and instructions planted inside the input. Jev applies a stated rule at 0.924; every open model tested (untrained, Tev1, Jevify Tier 2) at 0.62–0.70. Fine-tuning made the 4Bs notice a missing answer (0.48 → 0.66–0.70) and resist planted instructions (hijack rate 0.40 → 0.14–0.16).
• Retraining the 4B Tier 2 the Tev1 way did not help. Together's generated data and shuffled options, alone and together, moved held-out accuracy by −0.003 to −0.015 and left the phishing collapse in place; a fine-tune's behaviour outside jev-bench varies between identical runs far more than its jev-bench score does (results/c/). The phishing collapse is a moved threshold: one log-odds shift fitted on sixteen labelled emails takes Tev1 from 0.51 to 0.76 and the Jevify 4B Tier 2 from 0.56 to 0.70 on the emails it never saw (results/phishing-recalibration/).
• On three Jev benchmarks written by other people, an untrained 12B beats Jev by twenty points on phishing. Gemma-4-12B at Tier 0: 0.825 on 2,000 phishing emails against Jev's 0.628 (McNemar p = 10⁻⁸⁷); the untrained Qwen3.5-4B readout: 0.700 at ECE 0.023. Records identical, all 2,087, to the inputs Together AI used. Details: results/community/.
• Fine-tuning broke both fine-tuned 4Bs on phishing — Together's Tev1 and our own Tier 2. They flag 0.6% and 6.1% of the emails where their shared base flags 23.6% (16.3% in Tev1's own prompt, so the prompt is not the cause).
• Tev1 on jev-bench: 0.703 macro, ECE 0.086 (read in its own prompt, recipe fitted on validation; answers identical to Together's hosted model on 2,086 of 2,087 shared items). The fine-tune adds +0.030 over its base at equal prompt, almost all on the four tasks it trained on (+0.003 held-out); Tev1's prompt format alone lifts the untrained base +0.044 on the held-out sources. Details: results/tev1-comparison/.
• CLM-v0.1-8B (Stanford / NVIDIA contrastive model), served by its own code: 0.340 macro, ECE 0.338 — near chance on knowledge and many-option configs; probed by hand before reporting. Details: results/clm-8b/.
• Perplexity's pplx-decider-v1-27b on jev-bench: 74.8% macro accuracy, ECE 0.080, 84.0% on the six held-out tasks (their hosted API, as served; all 22,773 test records, no errors). It beats Jev overall (+1.6 points, interval [+1.1, +2.0]) and ties it on the held-out tasks, and it is better calibrated than Jev as served, but it is the least coherent system measured (sure loss 0.219 against Jev's 0.081; coherence-penalty fine-tunes reach 0.02-0.03). Its card's "JevBench public hard" row is a different benchmark of the same name. Details: results/pplx-decider-v1-27b/.
• An open 4B is now above Jev on the overall number. Qwen3.5-4B with LoRA + residual heads (Tier 2): macro accuracy 0.747 vs 0.733, ECE 0.110 vs 0.113, Noul 0.903 vs 0.881, TVD to human distributions 0.347 vs 0.432. The qualifier: on the six sources it never trained on, Jev leads 0.835 to 0.769. One seed.
• An open 4B now matches or beats Jev on every macro number. Qwen3.5-4B, Tier 2 at LoRA lr 3e-5: accuracy 0.734 vs 0.733, ECE 0.096 vs 0.113, TVD to human distributions 0.337 vs 0.432. Jev still leads the six held-out sources (0.835 vs 0.765). Published as Praveenrajus/jevify-qwen3.5-4b-t2-lowlr.
• At 9B, a frozen backbone with heads beats Jev where it trained. Three seeds with the six-epoch cap: held-out 0.743 ± 0.002, trained-source 0.703 / ECE 0.064 vs Jev's 0.694 / 0.122.
• Our recipe fitter read the rounded wire format and zeroed real high-K probabilities; fixed and every Tier 0 run refitted. The Gemma-4 instruct models were the ones it hurt (12B 0.149 → 0.086, E4B-it 0.148 → 0.092, E2B-it 0.158 → 0.123); nothing else moved by more than 0.003. Details: results/recipe-k/.
• Seed variance, measured — and explained. Five seeds of the Qwen3.5-2B residual head: held-out accuracy 0.579 ± 0.049 (two seeds match Tier 0's 0.628, three regress by ~0.09), trained accuracy 0.635 ± 0.007. The earlier claim that the residual "erases" the held-out regression was one seed at the top of that spread. The weight the head learns on the model's own log-score separates the basins (0.95 for the seeds that held up, 0.92–0.93 for the ones that collapsed; the same ordering over five 4B seeds), and capping training at six epochs collapses the spread to 0.621 ± 0.004 with the trained gain intact. At 4B over five seeds the trained gain (+0.047) is robust and the held-out gain is not (0.715 ± 0.025 vs Tier 0's 0.714). Per-seed results: results/qwen35-2b-t1r-s0 … -s4, results/qwen35-2b-t1r-e6-s0 … -s4, results/qwen35-4b-t1r-s1 … -s4.
• Jevified open models beat Jev on calibration and on human agreement. Qwen3.5-4B with residual heads reaches macro accuracy 0.698 against Jev's 0.733, ECE 0.089 vs 0.113, TVD to human label distributions 0.360 vs 0.432, and Score accuracy 0.507 vs 0.503. Qwen3.5-2B reaches ECE 0.069, the best of anything tested.
• LoRA (Tier 2) buys accuracy the frozen backbone cannot, and the learning rate decides its calibration — three seeds per arm. Qwen3.5-2B goes 0.632 → 0.690 macro accuracy (LoRA lr 3e-5) and the held-out ordinal scale gains +0.17 with better calibration. At 3e-5 the model is better than at 1e-4 on the six sources it never trained on — accuracy 0.716 ± 0.003 vs 0.700 ± 0.002, ECE 0.083 ± 0.004 vs 0.104 ± 0.014 — and its agreement with human label distributions (TVD 0.339) is the best of any model here (Jev 0.432). What the seeds took away: the low learning rate does not rescue calibration on ambiguous items — ChaosNLI ECE spans 0.16–0.33 across seeds in both arms, at Jev's 0.222 and far from the Tier 1 residual's 0.077 — and training on human label distributions did not help (TVD 0.433). Held-out accuracy under LoRA is seed-stable (±0.003) where the frozen-backbone head was bimodal (±0.049). Every 2B LoRA run overfits after one or two passes. Per-seed results: results/qwen35-2b-t2{,-s1,-s2}, results/qwen35-2b-t2-lowlr{,-s1,-s2}.
• Held-out sources changed the architecture. Heads that replace the model's own scorer gain +0.077 accuracy on trained sources and lose −0.098 on held-out ones; a zero-initialized residual on that scorer keeps the gain and erases the regression. Holding out records instead of whole sources would have hidden it.

Full catalogue with evidence and caveats: docs/FINDINGS.md.

Published models

Load any of them with load_jevified("<repo>") from the engine; each serves as a drop-in for the TypeSafe SDK (jevify-serve --model <repo>).

Jev 1.13.0 baseline

jev-1.13.0 on every test record: crisp, grounded decisions land in the accurate-and-calibrated corner; ordinal ratings and anything humans disagree about do not.

Every test record, one request each, scored by jevify-run. Predictions, full metrics with reliability bins, and figures live under results/<model>/. Columns: accuracy, top-label ECE, Brier, NLL, selective accuracy at 90% / 50% coverage, AURC, RPS and MAE (ordinal), AUROC (noul), total variation distance to the human label distribution. NLL is inflated on high-K configs because the API rounds probabilities to 0.01; read Brier and ECE as the proper scores.

Per-config table (22 configs, 14 metrics)

source	prim	n	acc	ECE	Brier	NLL	sel@90	sel@50	AURC	RPS	MAE	AUROC	TVD→human
arc_challenge	choice	1000	0.979	0.010	0.037	0.19	0.993	0.996	0.005
banking77	choice	1000	0.796	0.095	0.317	2.10	0.839	0.964	0.072
boolq	noul	1000	0.917	0.021	0.061	0.21	0.953	0.990	0.017			0.978
chaosnli	choice	1599	0.615	0.222	0.583	1.71	0.629	0.691	0.279				0.332
civil_comments	noul	2000	0.729	0.045	0.183	0.55	0.752	0.837	0.153			0.829	0.289
clinc150	choice	1000	0.893	0.033	0.159	0.82	0.941	0.986	0.024
fever_evidence	noul	1000	0.972	0.028	0.025	0.11	0.986	0.996	0.006			0.991
go_emotions	choice	1000	0.282	0.384	1.040	9.89	0.294	0.370	0.590				0.677
helpsteer2_helpfulness	score	1000	0.363	0.232	0.812	3.25	0.373	0.432	0.537	0.173	0.96
helpsteer2_verbosity	score	1000	0.341	0.231	0.793	1.86	0.368	0.424	0.594	0.132	0.72
ledgar	choice	1000	0.751	0.117	0.374	2.29	0.794	0.942	0.095
massive	choice	1000	0.808	0.090	0.295	1.92	0.859	0.968	0.056
measuring_hate_speech	score	1000	0.527	0.237	0.669	2.75	0.538	0.720	0.289	0.267	0.71		0.428
mmlu	choice	1000	0.923	0.027	0.124	0.46	0.960	0.986	0.021
mnli	choice	1000	0.883	0.032	0.176	0.50	0.917	0.982	0.035
paws	noul	1000	0.846	0.040	0.109	0.34	0.878	0.976	0.048			0.932
sms_spam	noul	800	0.965	0.068	0.035	0.15	0.985	0.993	0.010			0.976
sst5	score	1000	0.565	0.190	0.618	1.90	0.586	0.664	0.327	0.089	0.50
strategyqa_closed	noul	687	0.785	0.042	0.144	0.45	0.820	0.910	0.091			0.885
strategyqa_grounded	noul	687	0.956	0.059	0.038	0.15	0.984	1.000	0.004			0.994
stsb	score	1000	0.538	0.119	0.594	1.31	0.554	0.626	0.333	0.072	0.55
yelp5	score	1000	0.685	0.174	0.483	1.85	0.707	0.792	0.202	0.064	0.35

Reliability diagrams — stated confidence vs observed accuracy per config. Flat lines mean the confidence carries no information.

Model vs human probability on the calibration-gold configs — the axis that separates a decision model from a classifier.

Does its confidence know when humans disagree? On ChaosNLI (100 annotators per item) Jev's confidence is flat — 0.81 / 0.84 / 0.83 / 0.88 as human agreement goes from split (< 50%, n = 191) through contested (50–70%, n = 828) and clear (70–90%, n = 523) to consensus (≥ 90%, n = 57); Pearson r = 0.046 — while its accuracy on the same bands runs 0.47 / 0.55 / 0.73 / 0.96. It is more confident than the human majority on 80% of items.

Risk–coverage — the error rate a confidence-gated router actually gets at each coverage.

Performance vs decision-set size (cross-dataset, so difficulty is confounded — a hypothesis view).

Behavioral probes

Within-item experiments: the same state and gold answer, one factor changed. 200 items per source. Full write-up in results/jev-1.13.0/probes/README.md.

• Decision-set size is a cost, not a cliff: clinc150 99.5% → 91.0% from K=2 to K=151, banking77 99.5% → 82.0% at K=77, with ECE ≤ 0.10 and mean P(gold) tracking accuracy throughout
• Ambiguity is the cliff: GoEmotions is 85% at K=2 and 30% by K=25 (rater agreement with the plurality is only 0.66) while ECE climbs to 0.34 — confident and wrong
• Option order: argmax flips 0–2.5% on crisp small-K tasks, 4–7.5% on large-K routing, 13% on GoEmotions — modest, and scaling with ambiguity, not K.
• Opaque keys with descriptions kept: accuracy unchanged (banking77 0.82 → 0.81, clinc150 0.905 → 0.915, massive 0.815 → 0.805); without descriptions: chance. Jev reads semantics, not key strings.
• Distractor injection: ≤3.3% of probability mass leaks to nonsense options.
• Primitive geometry matters: the same yes/no question is better calibrated as Noul than as a 2-way Choice (BoolQ ECE 0.028 vs 0.054, Civil Comments 0.056 vs 0.126, at equal accuracy); Score beats an unordered Choice over the same levels (+2.5 points accuracy and lower ECE on sst5, yelp5 and helpsteer2).

Vision

The same three primitives, asked about an image. Records are built in memory from POPE (object hallucination, Noul), A-OKVQA (knowledge VQA, Choice K=4) and AI2D (science diagrams, Choice); the model is a vision-language checkpoint with no training, scored the same way as everything above, then given the Tier 0 recipe (a temperature per primitive and a Noul bias, fitted on validation splits only — POPE's is carved from its single split). Predictions and a text-only records manifest are in results/qwen3vl-2b/; the model is published as Praveenrajus/jevify-qwen3-vl-2b.

• The raw readout shows the text ordering (Noul better calibrated than Choice); the recipe erases it. After a per-primitive temperature the three sources sit within 0.02 ECE of each other. The raw gap measured how much a scalar could fix, not which primitive is easier.

• A starved vision encoder degrades safely. Holding records, prompt and decoder fixed and moving only the pixel budget: from 42 to 187 median image tokens, accuracy rises 0.048 and ECE falls 0.037. Losing accuracy while keeping confidence — confident hallucination — never appears at any budget tested. The budget also saturates silently (512 and 1024 patches both give 280 tokens), which only the measured token count reveals. Sweep data: results/vision-budget/.

• Served, an image question takes 68–79 ms on one A40 (~350 input tokens, ~85% of them image), four questions about one image 221 ms in one batched forward, 12.6 records/s batched — under Jev's 180–220 ms text round trip. results/latency/latency_vision.md.

• The vision tower is the wrong half to adapt. Three rank-16 LoRAs trained on the readout over 2,000 A-OKVQA records, confined to the tower, the decoder, or both, then recipe-fitted: the decoder adapter transfers to AI2D, a source it never saw (0.652 → 0.707, ECE 0.059 → 0.041; macro ECE 0.047 → 0.035), the tower adapter barely (0.669), and both together is not additive (0.697). After the decoder LoRA the fitted Choice temperature is 1.001: a proper scoring rule on the readout made the recipe redundant on the trained primitive. One seed per arm. results/qwen3vl-2b-t2-{vision,decoder,both}/, published as Praveenrajus/jevify-qwen3-vl-2b-t2.

Latency

Jev's round trip barely moves with the number of options. Probes against the API with the server's own clock (x-envoy-upstream-service-time) explain it as ~75 ms fixed + ~5.5 µs per input token, linear to 27,000 tokens, with option tokens costing exactly what state tokens cost — no cache, no fixed window, no batching wait — and an answer template filled in one pass rather than decoded (2,570 "output" tokens in 128 ms at K=255). Data and figure: results/jev-latency-probe/.

A Jevified model measured the same way on one A40: the research path pays ~120 µs per token; the same model through vLLM pays ~48 µs, halves the fixed floor and runs 4–5× more records per second — the same shape as Jev's curve, from throughput rather than from any option-specific mechanism. results/latency/.

At 8–12B. Qwen3-VL-8B answers an image question in 120 ms on one A40, Gemma-4-12B in 175 ms, Qwen3.5-9B in 170 ms (its hybrid layers run on reference kernels here) — all under Jev's text round trip. Qwen3.5-9B text through vLLM: 56–70 ms for K ≤ 27, 321 ms at K=151 (Jev 219). And on the same model and prompt, the readout costs the prefill while decoding the answer costs 52–106 ms per token on top: an eight-token answer is 4–7× the readout, a 32-token one 13–27×. results/latency/latency_8b*.md, latency_vision_*.md, latency_generate_*.md.

What is in this repo

Label audit and changelog

Every weak result was checked by reading samples of the model's errors. Verdicts, examples and the two v0.1.1 fixes that came out of it are in results/jev-1.13.0/README.md.

• v0.1.1 — helpsteer2_verbosity levels replaced with NVIDIA's verbatim length scale (v0.1 misdescribed them); go_emotions rebuilt from raw per-rater votes with soft labels; other configs unchanged.
• v0.1 — initial release.

Data v0.1.1, built with jevify-bench on 2026-09-20; results updated as they are measured.