1 · Jev is calibrated right up until humans disagree — which is when calibration matters. On ChaosNLI, where 100 annotators label every item, Jev's confidence is essentially flat no matter how much the annotators agree — 0.81 / 0.84 / 0.83 / 0.88 as agreement goes from a split (< 50%, n = 191) through contested (50–70%, n = 828) and clear (70–90%, n = 523) to consensus (≥ 90%, n = 57), Pearson r = 0.046 — while its accuracy on those same bands runs 0.47 / 0.55 / 0.73 / 0.96. It is more confident than the human majority on 80% of items; its distributions sit TVD 0.33 from the human ones, and 0.43 on hate-speech vote shares. Meanwhile it is excellent and well calibrated on crisp, grounded questions (ARC 0.979, FEVER 0.972, ECE ≤ 0.06).

2 · Evaluating on held-out records instead of held-out sources would have shipped the wrong architecture — and the head tells you itself when it has stopped generalizing. A trained head that replaces the model's own scorer gains +0.077 accuracy on sources it trained on and loses −0.098 on sources it never saw. Making it a zero-initialized residual on that scorer — so training provably starts at Tier 0 — keeps the gain (+0.080). The first version of this README said it also "erased the regression (−0.000)". Five seeds say otherwise: held-out accuracy is 0.579 ± 0.049, bimodal — two seeds match Tier 0 (0.628), three regress by ~0.09 — while trained-source accuracy is stable at 0.635 ± 0.007. What separates the basins is the weight the head learns on the model's own log-score: 0.95 for the seeds that held up, 0.92–0.93 for the ones that collapsed, and the same ordering at 4B (ρ = 0.90 and 0.97 within each backbone). Every extra epoch on the trained sources erodes that trust, early stopping on trained-source loss cannot see it — and capping the schedule at six epochs collapses the spread to 0.621 ± 0.004 with the weight at 0.965 for every seed and the trained gain intact. So Tier 1 is "+0.08 where it trained, Tier 0 where it did not, if it stops early", and the LM weight is the diagnostic. At 4B, five seeds: trained +0.047 every time, held-out 0.715 ± 0.025 against Tier 0's 0.714 — the +0.035 first reported was one seed at the top of the spread. At 9B, three seeds with the cap from the start: held-out 0.743 ± 0.002, and on the sources it trained on it beats Jev on accuracy and calibration (0.703 / ECE 0.064 vs 0.694 / 0.122) with the backbone frozen.

3 · An open 4B now matches or beats Jev on every macro number — and Jev still wins where it counts most. Qwen3.5-4B with LoRA and residual heads (Tier 2) at LoRA lr 3e-5: macro accuracy 0.734 vs Jev's 0.733, macro ECE 0.096 vs 0.113, TVD to human label distributions 0.337 vs 0.432, ChaosNLI ECE 0.211 vs 0.222. At lr 1e-4 the same model reaches 0.747 accuracy for a worse ECE (0.110). The qualifier is the split: it wins 9 of 22 configs on accuracy and 12 on ECE, not all 22, and on the six sources it never trained on Jev leads 0.835 to 0.765 — the open model's overall edge comes from the sixteen it did train on (0.722 vs 0.694). One seed each. Praveenrajus/jevify-qwen3.5-4b-t2-lowlr. Without touching the backbone, Qwen3.5-4B with residual heads alone reaches 0.698 / ECE 0.089 / TVD 0.360; the 2B head's ECE of 0.069 is the best of anything tested but, per (2), seed-dependent on unseen sources.

4 · Structure generalizes; knowledge does not. Heads trained with ≤16 options transfer unchanged to K=151 (clinc150 moves −0.008). What collapses under replacement is knowledge (arc_challenge −0.199, mmlu −0.119 even when trained) and ordinal scales never seen (measuring_hate_speech −0.482).

5 · LoRA buys the accuracy a frozen backbone cannot, and at 4B it buys calibration too. Tier 2 (rank-16 LoRA trained jointly with the residual heads, one A40) lifts Qwen3.5-2B from 0.632 to 0.690 macro accuracy and Qwen3.5-4B from 0.698 to 0.747, and fixes the Tier 1 weak spot outright: the held-out ordinal scale gains +0.17 accuracy and gets better calibrated. The learning rate decides the calibration, and three seeds per arm agree: at 3e-5 the 2B model is better than at 1e-4 on the sources it never trained on — accuracy 0.716 ± 0.003 vs 0.700 ± 0.002, ECE 0.083 ± 0.004 vs 0.104 ± 0.014 — at the same trained-source accuracy, and its agreement with human label distributions (TVD 0.339) is the best of any model tested (Jev 0.432). Two things the seeds took away: training on human label distributions did not help (TVD 0.433), and the low learning rate does not rescue the ambiguous-question calibration the LoRA costs — ChaosNLI ECE ranges 0.16–0.33 across seeds in both arms, at Jev's level (0.222) and far from the Tier 1 residual's 0.077. One thing they added: Tier 2's held-out accuracy is seed-stable (±0.003) where Tier 1's was bimodal (±0.049) — partly because every LoRA run stops after one or two passes, the regime in which (2) showed the head is stable too. At 4B the same learning-rate trade holds: 3e-5 gives held-out 0.765 / ECE 0.098 against 0.769 / 0.107 at 1e-4. And soft labels fail at the low learning rate as well (TVD 0.364 vs 0.332 for hard labels): the loss, not the schedule, is what makes them worse at matching human distributions. · detail

6 · Instruction tuning does hurt calibration — 1.3–2.1× worse raw ECE — but it is almost entirely a temperature problem. Gemma-4-E2B-it starts at ECE 0.370 and lands at 0.123 after one scalar per primitive; its accuracy is +0.195 over its base checkpoint for +0.008 ECE. Take the instruct checkpoint and always fit the temperature — on the exact distribution: our own fitter read the 4-decimal wire format, which zeroed real high-K probabilities and cost the Gemma models 0.03–0.06 ECE until it was fixed. · detail · detail

7 · What hurts Jev is ambiguity, not option count. Controlled within-item probes: with the gold answer always present, clinc150 goes 0.995 → 0.910 from K=2 to K=151, while GoEmotions is 0.850 at K=2 and 0.300 by K=25. Option order flips 0–13% of answers, scaling with ambiguity rather than K; opaque option keys cost nothing as long as descriptions remain; nonsense options attract ≤3.3% of the mass.

8 · The primitives are different instruments. The same yes/no question is ~2× better calibrated asked as a Noul than as a two-option Choice (ECE 0.028 vs 0.054); Score beats an unordered Choice over the same levels. This is why Tier 1 gives Noul its own absolute head instead of a softmax over {yes, no}.

9 · Two of our own benchmark bugs, found by reading the model's errors. HelpSteer2 verbosity levels contradicted NVIDIA's verbatim scale, and GoEmotions' single-label subset hid rater disagreement (rebuilt from raw votes; plurality agreement is only 0.66, which is the accuracy ceiling). Low benchmark scores deserve an audit before they become claims.

10 · It transfers to vision — and a picture question answers in under 80 ms. Qwen3-VL-2B, no training, on POPE / A-OKVQA / AI2D (4,244 records): a recipe fitted on validation splits takes macro ECE from 0.128 to 0.047 (POPE 0.046, A-OKVQA 0.037, AI2D 0.059) at unchanged accuracy. The raw readout shows the text ordering — Noul better calibrated than Choice — and the recipe erases it: the raw gap measured what a temperature could fix, not which primitive is easier. Starving the vision encoder costs accuracy and calibration together (42 → 187 image tokens: +0.048 accuracy, −0.037 ECE); the dangerous regime — losing accuracy while keeping confidence — never appears, and the budget saturates silently (512 and 1024 patches both give 280 tokens). Served through the same ask path as the text models, an image question takes 68–79 ms on one A40 (~350 input tokens, 85% of them image), four questions about one image 221 ms, 12.6 records/s batched — under Jev's 180–220 ms text round trip, and Jev takes no images. And the vision tower was the wrong place to look: a LoRA trained on the readout over 2,000 A-OKVQA records transfers to science diagrams it never saw when it lives in the decoder (AI2D 0.652 → 0.707, macro ECE 0.047 → 0.035) and barely when confined to the tower (0.669); after the decoder LoRA the fitted Choice temperature is 1.001 — training on a proper scoring rule made the recipe redundant on the trained primitive. Published: Praveenrajus/jevify-qwen3-vl-2b (Tier 0) · Praveenrajus/jevify-qwen3-vl-2b-t2 (decoder LoRA). · detail

11 · Jev's flat latency curve is arithmetic, not architecture — and a serving path gives our engine the same shape. Jev's round trip barely moves with the number of options (186 ms at K=4, 219 ms at K=151). Controlled probes with the server's own clock say why: ~75 ms fixed + ~5.5 µs per input token, linear to 27,000 tokens, with option tokens costing exactly what state tokens cost — no cache, no fixed window, no batching wait. 151 options are ~1,300 tokens, ~7 ms. The answer is a template filled in one pass (2,570 "output" tokens in 128 ms), not a decode. Our research path paid ~120 µs per token on an A40, so the same options cost it 150 ms; the same model on the same GPU through vLLM pays ~48 µs, halves the fixed floor and runs 4–5× more records per second. Below K≈60 a 2B on our A40 now answers in ~46 ms — 4× under Jev's round trip. · probes · detail

The original ladder (research path, four sizes, three tiers) is kept as results/latency/latency_ladder.png; Jev's server-time fit is results/jev-latency-probe/server_time_vs_tokens.png.

12 · At 8–12B the picture holds, and the speed claim gets its honest form. Qwen3-VL-8B, Qwen3.5-9B and Gemma-4-12B on the vision configs, no training: macro accuracy 0.84 (2B: 0.78), AI2D 0.75–0.77 (2B: 0.65), macro ECE 0.032 for the 8B after a recipe — and Qwen3.5-9B's raw readout is already calibrated (ECE 0.045 before any temperature), the only checkpoint tested for which that is true. An image question answers in 120 ms (Qwen3-VL-8B) to 175 ms (Gemma-4-12B) on one A40, still under Jev's 180–220 ms text round trip; through vLLM the 9B text model sits at 56–70 ms for K ≤ 27. The comparison that matters: on the same model and prompt, the readout costs the prefill (85–180 ms) while decoding the answer as text costs 52–106 ms per token on top — an eight-token answer is 4–7× the readout, a 32-token one 13–27×. Generation pays for tokens a typed decision never asked for; that is the saving, at 2B or 12B. · detail

13 · On benchmarks we did not write, an untrained 12B beats Jev by twenty points — and fine-tuning broke both fine-tuned 4Bs, ours included. Three public Jev benchmarks written by other people, scored unchanged (identical, all 2,087 records, to the inputs Together AI used): on 2,000 phishing emails, Gemma-4-12B with zero training is 0.825 against Jev's 0.628 (McNemar p = 10⁻⁸⁷), and the untrained Qwen3.5-4B readout is 0.700 at ECE 0.023. Together's Tev1 and our own Jevify Tier 2 — two independent fine-tunes of that same Qwen3.5-4B — flag 0.6% and 6.1% of the emails where their base flags 23.6%, and land near chance. On jev-bench, Tev1 (read in its own prompt, answers identical to Together's hosted model on 2,086 of 2,087 items) is 0.703 macro, ECE 0.086 with a recipe; its fine-tune adds +0.030 at equal prompt, all on tasks it trained on, while its prompt format alone lifts the untrained base +0.044 on the held-out sources — +0.129 on the held-out hate-speech scale. CLM-v0.1-8B, served by its own code, is 0.340. · Tev1 · community benchmarks · CLM

14 · Given a rule, Jev applies it; no open model does yet — and retraining ours the Tev1 way does not change that. Three new tests: 885 human-written legal fact patterns with the rule written into the question, "none of the above" with the right option removed, and instructions planted inside the input. Jev applies a stated rule at 0.924; every open model — untrained, Tev1, our Tier 2 — sits at 0.62–0.70, its largest lead anywhere in this project. Fine-tuning did make both 4Bs notice a missing answer (0.48 → 0.66–0.70) and resist planted instructions (hijack rate 0.40 → 0.14–0.16). Retraining our 4B Tier 2 with Together's generated data and shuffled options moved held-out accuracy by −0.003 to −0.015, left the phishing collapse in place, and showed that a fine-tune's behaviour outside jev-bench varies between identical runs far more than its jev-bench score does. That collapse is a moved threshold: one shift fitted on sixteen labelled emails takes Tev1 from 0.51 to 0.76 and our Tier 2 from 0.56 to 0.70 on the emails it never saw. · new tests · retraining · the phishing threshold

15 · Post-training shapes a decision readout stage by stage — and a coherence penalty removes the incoherence that fine-tuning adds. Scored at every published stage of five families, base checkpoints range from under- to over-confident, SFT leaves the readout close to calibrated, and preference optimisation (Tülu DPO, SmolLM3 APO — two labs, two algorithms) then makes it over-confident at flat accuracy (fitted temperatures up 1.3–2.4×). The first post-training step makes decisions more invariant in all eight base → post-trained pairs, and Gemma-4's instruct models score below their bases without their chat template. Readout fine-tuning — training a model on its own decision distribution — takes Qwen3.5 2B/4B/9B and Gemma-4-E4B to 0.70–0.76 accuracy at ECE 0.05–0.06 (Jev 0.733 / 0.113), but makes their answers to related questions contradict each other more (sure loss 0.15 → 0.28 at 4B). Adding that sure loss as a penalty cuts it tenfold, to 0.02–0.03 — below Jev's 0.081 — at no accuracy cost and with closer agreement to human label distributions at every size. Starting from the base checkpoint ends in the same place in Qwen3.5 (4B: 0.741 vs 0.743; Gemma-4 closes most of its wider gap), and full fine-tuning matches LoRA at 2B. · results · FINDINGS 17–18

• jev-bench v0.1.1, Jev 1.13.0 baseline, figures, label audit, behavioral probes
• Tier 0 engine + server + offline recipe search
• Tier 0 sweep: 9 open checkpoints scored on all 22,773 test records against Jev
• Tier 1 residual decision heads, with held-out-source generalization measured
• Findings, figures and reports published (docs/FINDINGS.md)
• Published models + hosted playground and API
• Tier 1 replicated on a second backbone (Qwen3.5-4B)
• Training portable off Modal (python -m jevify.train)
• VLM backbones: Tier 0 on POPE / A-OKVQA / AI2D; vision tower inspectable, budgetable, freezable, LoRA-scopable
• Vision-encoder budget sweep: accuracy and calibration degrade together, budget saturation measured
• Tier 2: LoRA jointly with residual heads — +0.058 macro accuracy at 2B; at lr 3e-5 the best held-out ECE and human-agreement of any model; overfits after one pass
• Tier 2 ablation: learning rate vs hard-label loss — the learning rate was the cause of the overconfidence; soft-label training is a negative result on human agreement
• Latency: Jev's server-time law measured; vLLM serving path with the same numbers, 4-5x throughput
• Five-seed error bars on Tier 1: held-out generalization is seed-dependent (0.579 ± 0.049); README corrected
• The LM-weight diagnostic and the epoch-cap intervention (0.621 ± 0.004); 4B replicated over five seeds
• Tier 2 at 4B: macro 0.747, above Jev's 0.733; Jev still leads on held-out sources — published as jevify-qwen3.5-4b-t2
• Vision served at Jev speed: recipe-fitted Qwen3-VL-2B published; 68–79 ms per image question on one A40
• Seeds on the Tier 2 arms: the learning-rate result holds (3 seeds each); LoRA held-out accuracy is seed-stable
• Soft labels at the low learning rate; the low learning rate at 4B
• Vision Tier 2: LoRA on the readout, tower vs decoder vs both — the decoder is where transfer lives (AI2D +0.055 held out)
• 8B-class study: Qwen3-VL-8B, Qwen3.5-9B, Gemma-4-12B — vision Tier 0, latency, readout vs generation on one A40
• Gemma-4-12B text Tier 0 (0.716, closest to Jev without training); Tier 1 at 9B (seed-stable, beats Jev on trained sources)
• Recipe fitter fixed (it read the rounded wire format); every Tier 0 run refitted; T(K) measured and left opt-in
• Tier 2 at the low learning rate on 4B (matches Jev's accuracy, better ECE and human agreement); soft labels fail at both rates
• Post-training study: five families at every published stage, one battery (FINDINGS 17)
• Readout fine-tuning + coherence penalty at 2B/4B/9B and on Gemma-4; full fine-tuning at 2B (LR swept on validation); 15 model repos published with reproduction checks
• Full fine-tuning at 4B (two GPUs per run, in progress)
• More ordinal scales; Decision 1.0 (vLLM Semantic Router) on jev-bench; synthetic structure data
• Label-first synthetic data pipeline; HF Space; model zoo