rsi-jev-v2.0-qwen3.5-2b

A typed-decision model: Qwen/Qwen3.5-2B-Base with its tower fine-tuned and a trained option-scoring head. It answers choice / noul / score questions about a document in one forward pass, and speaks the Jev HTTP API.

This checkpoint is seed 17, fixed in advance as primary rather than chosen for scoring best:

Re-computed from these exact files after upload-time reload. Kernel stack fla-0.5.2/torch-2.7.1+cu128; full recipe in meta.json.

import sys; sys.path.insert(0, 'code')   # after snapshot_download of this repo
from load_release import load_release
model, tok, enc, meta = load_release('.', device='cuda')
print(meta['calibration'])   # the fitted calibration this checkpoint ships, or 'none'

load_release applies calibration.safetensors when the checkpoint carries one, so the probabilities are the calibrated ones this card reports. A loader that skips those files returns the raw distribution instead: the same chosen option, different numbers.

Everything below is the version record from the project repository, unchanged.

---

RSI-Jev v2.0

Describes release v2.0 · updated 2026-09-26

One typed-decision model, 2B. Given a document and a set of typed questions — yes/no, pick-one-of-k, rate-on-a-rubric — it returns a calibrated probability for every option in one forward pass, with no generated text.

Two things changed from v1.0. It is trained on the train splits of the decision benchmarks it is measured on, which v1.0 never touched. And it carries a fitted confidence head, so the probability it reports means something closer to what it says.

Read section 4 before quoting anything here. On the twelve-benchmark suite v2.0 scores 0.705 against the v1.0 recipe's 0.609. On three benchmarks held out from the entire search it does not beat v1.0. Both facts are measurements of the same model, and the second is the one that answers "is it a better model".

---

1. The task

Unchanged from v1.0. A typed decision is a document (the state) plus one or more questions, each of a declared type:

Pooled top-1 is the fraction of decisions whose highest-probability option matches the teacher's modal label. ECE is expected calibration error: bin decisions by the confidence the model reported, compare each bin's confidence to its accuracy, and average the gaps by bin size. Lower is better, 0 is perfect.

2. How the model is built and trained

The architecture is v1.0's, unchanged, plus the confidence head in §2.1. What changed is the data.

The training data, in full:

This is the difference that matters, and it cuts both ways. v1.0's headline was a zero-shot number: it had never seen a typed-decisions document. v2.0's is not. Ten of the twelve suite benchmarks contributed training data, so v2.0's score on them measures something v1.0's did not.

What is still never trained on, checked rather than asserted: every benchmark's test split, verified by the arm runner; the three held-out benchmarks in §4.3; and the calibration data below. No training document shares a state with any test split.

2.1 The confidence head

A linear head predicting one positive temperature per question, which divides that question's logits. Because every option in a row is divided by the same number, the answer never changes — only how sharp the probabilities are. It reads features of the forward pass that already happened (the option distribution's shape, the question type and option count, and a 16-dimensional projection of the decision hidden state), so a decision is still one forward pass.

It is fitted after training, on data withheld from it: 20% of the typed-decisions train cases, 3% of synth, 5% of the replay, and the dev partitions of the suite train splits — 1,576 cases in total were withheld from v2.0's training so the head had something honest to fit on. Score questions are allowed to soften but never to sharpen, which costs a little calibration and keeps the model's score distributions faithful to the teacher's spread.

3. Checkpoints

v2.0's typed-decisions figure is not comparable to v1.0's: v2.0 trained on that benchmark's train split. Its ECE is comparable, and is the honest improvement.

There is no v2.0 at 0.8B. The recipe was only ever run at 2B.

On reproducing its training run. Re-scored from disk with this repo's code, the checkpoint reproduces the training run's per-question predictions at 1.0000 on all three verification targets — typed-decisions in both option orders, and MMLU-Pro at 0.3050. That is the check v1.0's card requires, and it passes.

It is worth knowing what it took to say that cleanly. The same check on a different GPU architecture returned 0.9980 on MMLU-Pro — two questions in a thousand — and 0.3060 rather than 0.3050. Five MMLU-Pro questions have a top-2 logit margin below 0.001, the narrowest 0.0005, and eighteen are below 0.01 against a median of 0.383; reordering an fp32 sum across GPU architectures moves a logit by about that much. So a couple of MMLU-Pro answers are decided by arithmetic order rather than by the model, and MMLU-Pro on this checkpoint should be read as 0.305 ± 0.001. typed-decisions has no margins that thin and reproduces exactly on every machine tried. The bf16 tower, offered for speed and used for no published number, agrees at 0.9935 / 0.9985 on typed-decisions and 0.9880 on MMLU-Pro.

Every claim above is a file in the checkpoint: verify.json is that re-scoring, verify_blackwell.json the second machine, roundtrip_200.json the reload check on 200 questions with the calibration applied (200/200, largest probability difference 2.9e-4), and MANIFEST.json the sha256 of every file so a download can be checked.

A v2.0 checkpoint ships calibration.safetensors and calibration.json beside the weights. load_release.py applies them automatically; without them the model loads uncalibrated and its ECE is 0.0893, not 0.0546.

Apache-2.0, following the base model, as v1.0 is.

4. Results

4.1 The twelve-benchmark suite

Four-seed means of each recipe, same evaluator, same kernel stack. The suite mean is weighted; the weights are in ../BENCHMARKS.md.

Confirmed, not just measured. The v2.0 recipe was repeated on four seeds it had never been run on, against the previous champion run on those same seeds: 0.7029 against 0.6337, +0.069, higher on all four. Over all eight seeds the gain is +0.0708 and MMLU-Pro is −0.0272.

Ten of these twelve benchmarks contributed train-split data. The two that did not are MMLU-Pro, which fell, and procedural, which also fell. That pattern is the result.

4.2 Calibration

On typed-decisions' 2,000 test decisions, binned by the confidence v2.0 reported:

This is the part that is worth having. v2.0 says 0.9 or better on 539 of the 2,000 decisions and is right on 98.3% of them — nine mistakes. v1.0 reached 0.9 on only 29 decisions out of the same 2,000, so there was almost nothing to act on: it was right when it was confident, and it was hardly ever confident. Over a quarter of this workload can now be decided automatically and the rest routed to something slower.

v1.0's ECE on this split was 0.0978 against v2.0's 0.0678, and the confidence head fixes both directions of the error: the model was under-confident on typed-decisions and over-confident out of distribution.

4.3 The held-out comparison

Three benchmarks were frozen before this work started and read exactly once, after the release model was chosen. Neither model trained on them, so this is the only place where v1.0 and v2.0 are measured on equal terms.

v2.0 is not a better decision model than v1.0. It is flat on two of the three and worse on the third. The suite gain of +0.096 is very largely the benefit of having trained on those benchmarks' own data, and it does not travel.

Calibration does travel, partly: ECE improves on scienthoon (0.0824 → 0.0599) and on MMLU-Pro (0.1009 → 0.0399), and gets worse on tasksource (0.0460 → 0.0888), where v2.0 is more confident than v1.0 without being more accurate.

What to use which for. If your documents look like the suite's benchmarks, v2.0 is much stronger on them. If they do not, v1.0 is as good and v2.0's confidence may be optimistic. If you want calibrated probabilities on typed decisions, v2.0.

5. Speed

Unchanged by anything in this release: the confidence head is a dot product on features already computed, and adds no forward pass.

Reading the document dominates, which is why a batch of questions about one document is the shape to ask in. Measure it on your own machine with scripts/bench.py → ../BENCHMARKS.md.

6. Acting on the probability

The routing table below is v2.0 on typed-decisions: act on the most confident x% of decisions and defer the rest.

The confidence field is not the probability

The API returns both, and they are different numbers. The probability is the model's mass on an option. confidence is the peak statistic Jev documents, (K * p_max - 1) / (K - 1) — for three options, "(3 x largest probability - 1) / 2". It measures how peaked the whole distribution is, is 0 at uniform and 1 at one-hot, and is absent from noul answers.

What v2.0 changes is the input to that formula, not the formula. The confidence head rescales the probabilities, so confidence moves with them. Route on the probability of the option you intend to act on, as §4.2 and the table above do; use confidence when what you want to know is whether the model is spread across options.

Because v2.0's probabilities are better calibrated, a threshold chosen on one document set transfers better than it did with v1.0 — within the caveat in §4.3, which is that on unfamiliar documents v2.0 is the more confident model without being the more accurate one.

7. What it costs to run

Unchanged from v1.0 §7; the weights are the same size and the calibration files add 140 KB.

8. Fixed after release

Nothing yet.

9. Limitations

• The headline does not generalise. §4.3. On benchmarks it was not trained for, v2.0 is flat to slightly worse than v1.0.
• Its benchmark scores are not zero-shot. Ten of the twelve suite benchmarks are in its training data, by train split. Do not compare them with any model's zero-shot numbers, including v1.0's.
• General knowledge got worse. MMLU-Pro 0.305 against v1.0's 0.3550. Fifteen percent general-knowledge replay held most of the loss back; raising it to 30%, or weighting it toward science, did not help further.
• Procedural decisions got worse: 0.6223 against the v1.0 recipe's 0.647.
• More confident on unfamiliar documents. Mean confidence rose on every held-out benchmark while accuracy did not, so the calibration gain is a property of the domains the head was fitted on.
• Score questions stay slightly under-confident, deliberately: the floor that protects their distributions costs about 0.010 of suite ECE.
• Checking the suite needs upstream checkouts. The loader ships (rsijev/targets_suite.py, scripts/suite.py --list) and every split is pinned to a public upstream commit, so the headline is reproducible — but you fetch four or five repositories yourself, because this repo redistributes none of them. Two of the three held-out benchmarks come from upstreams that have since gone private and survive only as vendored copies inside the Kev repo.
• Its corpus can be assembled from the repo, but two of its inputs cannot be downloaded yet. The four builders that assemble it now ship (../rsijev/README.md). What is missing is data: the replay pool's source files, which came from an external task registry, and the frozen suite definitions that the train-split conversion decontaminates against. v1.0 needed only scripts/build_synth_corpus.py, so v2.0 is a step harder to check, and closing it is a data release rather than a code one.
• One model, one size. 2B only, and one seed's checkpoint.

10. How it got here

v1.0 came out of five turns that each ruled something out:

Then about fifty more arms, of which four were kept: the benchmark's own train split with general-knowledge replay, the other benchmarks' train splits, and two forms of the confidence head. Every training-objective arm and every reinforcement-learning arm was rejected — including five separate RL rewards on the decision itself, and a reasoning teacher that scores 0.677 zero-shot but disagrees with the benchmark's teacher precisely on the ambiguous items.

The chain that produced this release, out of those fifty:

The other forty-odd directions were rejected. What each one was, what it scored and why it failed is the search record, not this release's record, and it is in ../EXPLORE.md.

One bar was moved after seeing a result, and it should be read with that in mind. The recipe that became v2.0 first failed the MMLU-Pro guard at −0.026 against a 0.020 limit. The run owner widened that limit to 0.030 on the grounds that MMLU-Pro is a guard against forgetting rather than a target. The original rejection is still in the log.