rsi-jev-v2.1-qwen3.5-2b

A typed-decision model: Qwen/Qwen3.5-2B-Base with its tower fine-tuned and a trained option-scoring head. It answers choice / noul / score questions about a document in one forward pass, and speaks the Jev HTTP API.

This checkpoint is seed 17, fixed in advance as primary rather than chosen for scoring best:

Re-computed from these exact files after upload-time reload. Kernel stack fla-0.5.2/torch-2.7.1+cu128; full recipe in meta.json.

import sys; sys.path.insert(0, 'code')   # after snapshot_download of this repo
from load_release import load_release
model, tok, enc, meta = load_release('.', device='cuda')
print(meta['calibration'])   # the fitted calibration this checkpoint ships, or 'none'

load_release applies calibration.safetensors when the checkpoint carries one, so the probabilities are the calibrated ones this card reports. A loader that skips those files returns the raw distribution instead: the same chosen option, different numbers.

Everything below is the version record from the project repository, unchanged.

---

RSI-Jev v2.1

Describes release v2.1 · updated 2026-09-27

One typed-decision model, 2B. Given a document and a set of typed questions — yes/no, pick-one-of-k, rate-on-a-rubric — it returns a calibrated probability for every option in one forward pass, with no generated text. The task, the architecture and the confidence head are v2.0's, unchanged.

v2.0 had to be published with a caveat: its suite gain came very largely from training on the suite's own train splits, and on three benchmarks frozen before the search began it did not beat v1.0. v2.1 is the version where that caveat goes away. It is ahead of v1.0 on all three of those held-out benchmarks, significantly so on two, and it recovers the general knowledge v2.0 lost.

---

1. The task

Unchanged from v1.0 and v2.0: a document (the state) plus one or more typed questions — noul, choice, score — and a probability per option, returned in one forward pass. Pooled top-1 is the fraction of decisions whose highest-probability option matches the teacher's modal label; ECE is expected calibration error, the average gap between the confidence a model reports and how often it is right at that confidence, weighted by bin size, lower being better.

2. How the model is built and trained

The architecture is v1.0's and the confidence head is v2.0's, both unchanged; see v2.0 §2. Three things changed, all in the data and the optimiser, and none of them in how inference works.

1. The procedural benchmark's train split joined the corpus (3k cases). v2.0's corpus did not have it, and procedural was one of the two benchmarks v2.0 got worse at.
2. The general-knowledge replay was weighted toward science, and half of it recast to ten options so its shape matches MMLU-Pro's.
3. The bottom 8 of the tower's 24 layers train at one tenth the learning rate.

The third is the one worth reading twice. v2.0's single real regression was general knowledge: MMLU-Pro fell from v1.0's 0.355 to 0.305 while every decision benchmark rose. Five things were tried against it:

The first is the interesting failure. Holding the tower's language modelling close to the base model works as an objective — the retention KL stayed near 0.02 nats per token — and MMLU-Pro eroded anyway. What the decision readout uses is not what next-token prediction preserves, so protecting the latter does not protect the former.

The rest say that this was never a data problem. More and better-shaped replay moves MMLU-Pro by +0.008. A uniformly smaller learning rate moves it by nothing. Freezing the lower layers moves it by +0.016 and breaks three decision benchmarks, because those layers do need to adapt. Letting them adapt slowly gets more of the knowledge back than freezing them does and costs nothing at all: +0.054 on the recipe, and past v1.0. Fitting a decision objective through the whole tower at one rate was overwriting the representation the knowledge sat in.

3. Checkpoints

As with v2.0, the typed-decisions figure is not comparable to v1.0's, because both v2.0 and v2.1 trained on that benchmark's train split. Against v2.0, which trained on the same split, v2.1 is flat there (−0.002) and its gains are elsewhere.

2B only, one seed's checkpoint. There is no v2.1 at 0.8B.

Reproducing its training run. Re-scored from disk with this repo's code, the checkpoint reproduces the training run's per-question predictions at 1.0000 on all three verification targets — typed-decisions in both option orders, and MMLU-Pro. verify.json is that re-scoring, run on an H100.

The same check on a second GPU architecture returns 0.9990 on MMLU-Pro — one question in a thousand — and that is the more useful fact. verify_blackwell.json is that run, shipped beside the first rather than discarded, because it is the evidence for the ±0.001 below. Neither figure was chosen over the other; both are what the published code returned on the machine it ran on.

verify_bf16.json is the bf16 tower, roundtrip_200.json the reload check with the calibration applied, cached_agreement.json the check that the two serving paths — with and without the prefix cache — return identical probabilities (162 questions, 0 flips, largest difference 0.0), and MANIFEST.json the sha256 of every file so a download can be checked.

MMLU-Pro on this checkpoint should be read as ±0.001. Three of its thousand questions have a top-2 logit margin below 0.001 — the narrowest 0.00004 — and eighteen are below 0.01, against a median of 0.534. (Those margins are a property of the predictions, so load_release.py --ckpt DIR --verify regenerates them and you can recount.) Reordering an fp32 sum across GPU architectures moves a logit by about that much, so a handful of MMLU-Pro answers are decided by summation order rather than by the model. v2.0 §3 works the effect through in full.

The checkpoint ships calibration.safetensors and calibration.json; load_release.py applies them automatically. Without them the model loads uncalibrated and its probabilities are the raw ones, which match no number in this card.

Apache-2.0, following the base model, as v1.0 and v2.0 are.

4. Results

4.1 The held-out comparison — read this first

Three benchmarks were frozen before any of this work started, and are read once per release. Neither v1.0 nor v2.1 trained on them, so this is the only place the two are measured on equal terms. All three models were scored in one job, on one code path.

p is an exact two-sided McNemar test on the decisions where the two models disagree. Against v2.0 rather than v1.0 the same test gives +0.0398 (p=0.0033), +0.0357 (p=0.078) and +0.0836 (p=5.4e−10).

v2.1 is a better decision model than v1.0, not only a better suite score. SemIf is positive but 252 questions give it little power, so read it as consistent with the other two rather than as evidence on its own.

4.2 The twelve-benchmark suite

The recipe columns are seed means: v1.0's and v2.0's over four seeds as published in v2.0 §4.1, v2.1's over seeds 17 and 29, with a fresh seed 97 run afterwards as confirmation (suite 0.7286, MMLU-Pro 0.360). The paired gain over the v2.0 recipe is +0.0219 on seeds 17 and 29 and +0.0257 on seed 97 — same sign, and larger on the seed the recipe had never been run on.

Checkpoint against checkpoint, on the same suite and the same evaluator, v2.1's is 0.7291 against v2.0's 0.7056.

Where that +0.024 comes from, weighted: procedural +0.0100, MMLU-Pro +0.0078, the other ten benchmarks +0.0057 net. So three quarters of it is two benchmarks, and the two are different in kind. Procedural's +0.25 is what training on a benchmark's own train split buys — the same effect as v2.0's headline, and it should be read that way and not as a capability gain. MMLU-Pro's +0.078 is not: its train data is in no corpus here, so that one is transfer, and it is the change §4.1 measures.

4.3 Calibration

The confidence head is v2.0's, fitted the same way and in the same mode (oof_head_scorefloor); see v2.0 §2.1. Suite ECE is flat against v2.0 (0.0556 against 0.0546) — v2.1's improvement is accuracy, not calibration.

On typed-decisions' 2,000 test decisions, binned by the confidence v2.1 reported (python scripts/calibration.py --model v2.1-2b prints this table):

Acting on the most confident x% of those decisions and deferring the rest: 100% → 0.790, 80% → 0.853, 60% → 0.893, 40% → 0.950, 20% → 0.993 — scripts/routing.py is the same measurement, and ../BENCHMARKS.md says how to run both.

On the held-out benchmarks calibration travels unevenly. ECE is the best of the three models on scienthoon (0.0824 → 0.0599 → 0.0542) and recovers most of v2.0's tasksource regression (0.0460 → 0.0888 → 0.0614), but on SemIf it is worse than either (0.0943 → 0.0916 → 0.1193) — there v2.1 is under-confident, reporting a mean 0.770 against 0.885 accuracy, which is the conservative direction but is still error.

5. Speed, the wire format, and what confidence means

All unchanged from v2.0, which is unchanged from v1.0: one tower pass per question, the calibration is a division inside the readout, and nothing generates text. 52 ms to read the document and 11.5 ms per decision on an NVIDIA GB10, bf16; reading the document dominates, so a batch of questions about one document is the shape to ask in. See v2.0 §5 and §7; the weights are the same size and the calibration files add 140 KB.

The API returns a probability and a confidence, and they are different numbers. The probability is the model's mass on an option. confidence is the peak statistic Jev documents, (K * p_max - 1) / (K - 1) — 0 at uniform, 1 at one-hot, absent from noul answers. Route on the probability of the option you intend to act on, as §4.3 does; reach for confidence when what you want to know is whether the model is spread across options.

6. Limitations

• Its benchmark scores are not zero-shot. Eleven of the twelve suite benchmarks now contribute train-split data — procedural joined the list this release. Do not compare those numbers with any model's zero-shot numbers, including v1.0's. The held-out three in §4.1 are the comparison that is fair.
• procedural's +0.25 is in-domain. It is the largest single number in this release and the least interesting one.
• General knowledge is recovered, not proven intact. 0.383 beats v1.0's 0.355 and v2.0's 0.305 on MMLU-Pro, and that is one benchmark of general knowledge measured as a typed decision. Fine-tuning a tower for a single-position decision objective destroys its language modelling outright — the tuned tower emits whitespace — so "does not forget" is a claim about this benchmark and nothing wider.
• SemIf calibration got worse, in the under-confident direction. §4.3.
• The corpus gate failed and was overridden. Adding procedural shifts the corpus's label mix away from the reference profile: first-option rate 0.278 against 0.507, and the score-level histogram moves (total variation 0.150), because procedural's rubrics have more levels than the rest of the corpus. The mix is the experiment here, so the gate was overridden deliberately and the failure recorded rather than tuned away.
• The held-out set is spent. Those three benchmarks have now been read for two releases. A held-out set is consumable, so v2.2 needs a freshly frozen one; see ../BENCHMARKS.md.
• Reproducing the suite needs upstream checkouts, and two of its corpus inputs are not downloadable yet. Unchanged from v2.0 §9.
• One model, one size, one seed.

7. How it got here

The chain from v2.0 to here, out of the arms run after it:

Rejected along the way, and worth knowing because each closes a direction: a 4B tower on v2.0's recipe (failed to train at all), the lr ×0.1 treatment at 0.8B (0.6834, discarded — the 0.8B line's own keeper is a different recipe at 0.6791, well below 2B), recasting all of the replay to ten options rather than half, raising the replay dose to 30%, eleven extra coverage sources, two round-2 data-selection policies head to head (no difference), and eleven more reinforcement-learning arms — four of them against a matched supervised control for the first time, and every one of those landed at or below its control. Counting every arm whose training signal was a reward rather than the teacher's target distribution, this project has now run eighteen and kept none. The full record is ../EXPLORE.md.

A rule was changed mid-search, and this release should be read with it in mind. The number of seeds required to confirm a candidate was changed from a fixed four to scaling with the effect size: one seed by default, a gap of about +0.015 or more confirmed by one fresh seed, smaller gaps by up to three plus a confirmation seed. v2.1's recipe was confirmed under the new rule — two seeds plus fresh seed 97 — not the old one. The reasoning and the original rule are in ../BENCHMARKS.md.