rsi-jev-v6.1-vl-4b

A typed-decision model built by a self-improving loop of AI agents. It is the weight average of v6.0-VL and a second fine-tune of Qwen/Qwen3.5-4B-Base, with decision heads at layers 16, 20 and 32: each question answers at the first depth that is confident enough. It answers choice / noul / score questions about a document and up to four images, and speaks the Jev HTTP API.

v6.1-VL is a System One model: one forward pass, no generated reasoning, a probability for every option. effort sets how many layers that one pass uses (16, 20 or 32), not how many tokens it writes; it writes none.

pip install "rsi-jev[fast,vision] @ git+https://github.com/Shanghua-Gao/RSI-Jev"
rsi-jev serve v6.1-vl-4b

Everything below is the version record from the project repository.

---

RSI-Jev v6.1-VL 4B

Describes release v6.1-VL · updated 2026-10-07

Two fine-tunes, averaged. v6.1-VL is the weight average of v6.0-VL and a second fine-tune of the same base, trained on other data. Nothing was trained after the average. Each exit's temperature was refit, and the exit policy was chosen again with v6.0-VL's rules. The architecture, the effort levels and the API are v6.0-VL's: decision heads at layers 16, 20 and 32 of Qwen3.5-4B-Base, and a question answers at the first one that is confident enough.

Against v6.0-VL on the same evaluation, accuracy is higher on the suite, the held-out set, MMLU-Pro, images and KoBBQ. On the public Decision Index 0.3 it scores 50.98 (v6.0-VL: 46.23). Calibration is worse: final ECE 0.048 against 0.036.

v6.1-VL is a System One model: one forward pass, no generated reasoning, a probability for every option. effort sets how many layers that one pass uses (16, 20 or 32), not how many tokens it writes; it writes none.

It is a 4B model, named as Qwen names the 4.66B Qwen3.5-4B-Base: 4.69B parameters, the same architecture as v6.0-VL — the 32 decoder layers (3.57B), the token embeddings (0.64B), the vision tower (0.33B), the main decision head (0.05B) and the two early-exit heads (0.10B). The checkpoint is self-contained.

Checkpoint: shgao/rsi-jev-v6.1-vl-4b · this record: versions/v6.1-vl.md

1. The task

Unchanged from v6.0-VL: a state (text, optionally up to four images), one or more typed questions, a probability for every option in one forward pass, and usage.depth, the layer each question was answered at.

2. How the model is built and trained

Full fine-tuning throughout (no LoRA), from Qwen/Qwen3.5-4B-Base, own-token option readout (specs in data/v6.1-vl_recipe/).

1. Member A is v6.0-VL (stages 1-4 of its record): a trunk with detached early exits trained 10,000 steps on a cleaned 148-source text corpus, the exit heads refit, 3,000 image steps, and all four heads retrained (seed 1).

2. Member B is a second line of the same model, five stages from the base:

   • text, 28,886 steps on a 95-source corpus, with the exit heads at 12, 16 and 20 trained together with the trunk;
   • images, 3,000 steps, on v6.0-VL's image corpus;
   • 3,000 steps on a mix of Decision Index training pools (35%), knowledge (15%) and replay (50%), 158 sources, exit heads detached from the trunk;
   • 3,000 steps on a second mix, 48,049 training questions: Decision Index training pools of two generations (35%), probe sets (10%), knowledge (15%), text replay (20%) and image replay (20%);
   • all four heads retrained, 600 steps, seed 1, as in v6.0-VL's stage 4.

   Member B on its own is not releasable: MMLU-Pro 0.418 at layer 32, below the 0.429 bar.

3. The average. Every tensor of the tower, the main head and the exit heads is the mean of the two members, weight 0.5 each. Nothing is trained.

4. Per-exit temperatures, refit on held-out development rows. v6.0-VL's temperatures do not transfer: averaging shrinks the logits, and at v6.0-VL's temperatures the average is underconfident. Development rows that either member trained on were dropped before the fit.

5. The exit policy, chosen again with v6.0-VL's rules (below). The layer-12 exit is fitted but not in the package, as for v6.0-VL.

The two members are close: the whole fine-tune chain moves the trunk by about 1.7% of its norm. The gain is in the tower. The averaged tower with v6.0-VL's own heads gives 98% of the averaged model's gain at layer 32. v6.0-VL's tower with averaged heads gives none of it. Uneven weights did not help: 0.3 / 0.7 is level with 0.5 / 0.5 within noise, and 0.7 / 0.3 is lower (−0.016).

Image sources and their licences. Both members trained on v5.0-VL's image corpus (member B on a 20% replay of it). The images are the original datasets', under their own terms. Five sources are non-commercial or research-only, and several others carry no clear grant for their images:

Text sources member B adds. Beyond the Decision Index training pools v6.0-VL already used (among them non-commercial ANLI and RAGTruth parts, and share-alike HoVer and ForecastBench), member B's last mix trains on:

The MMLU-Pro-style proxy questions were checked against the MMLU-Pro test set (exact, n-gram and embedding matches, the closest pairs read by hand): none is a test item or a rewording of one. The weights are released under Apache-2.0, as earlier releases were. Whether weights trained on non-commercial data inherit those terms is not settled; check the sources above against your use.

Exits. Each exit has its own scalar temperature, refit on the average: T = 0.87 at layer 16, 0.63 at 20, 1.02 at 32 (v6.0-VL: 1.88, 1.97, 1.65). The layer-32 value is in both calibration.safetensors and calibration.json.

Choosing the exit. With auto, a question answers at the first exit whose calibrated top probability reaches that exit's threshold, 0.85 at layer 16 and 0.50 at layer 20, else at layer 32. These were picked by v6.0-VL's rule on one half of the development sets and confirmed on the other, then read once on test. With no effort set, multi-question requests use one threshold at both exits, τ = 0.95. That value is the fallback of a rule written before the fit, and it is not confirmed: the rule's own pick, 0.85, missed the depth cap on the confirmation half (§4.2). Mean depth on the suite with auto: 19.5 layers.

3. Checkpoints

pip install "rsi-jev[fast,vision] @ git+https://github.com/Shanghua-Gao/RSI-Jev"
rsi-jev serve v6.1-vl-4b                    # default: single questions at 32, the 0.95 cascade on multi-question requests
rsi-jev serve v6.1-vl-4b --effort auto      # a server default; a request's own "effort" field wins

¹ Without open_jev_ood; the earlier rows include it. ² The suite's calibration is not compared across releases (v6.0-VL's record, §4.1); final-set ECE is in §4.1 below. v6.1-VL's suite and held-out figures are the bf16 package as served by default.

The package was checked on the paths users get. The low, medium and high effort paths give bitwise the forced-exit results. The unset default is bitwise the package without the auto thresholds, and a threshold of 1.0 is bitwise high. auto picks the same exit as the offline cascade on all 2,186 test questions, with probabilities within 4e-5. Served in bf16, the package scores 0.793 / 0.729 / 0.463 / final ECE 0.045 by default; the fp32 read below gives 0.793 / 0.728 / 0.462 / 0.048 at layer 32.

4. Results

4.1 Against v6.0-VL

Both models read once on the same evaluation sets with the same code, text in fp32; ECE at each model's served temperatures. KoBBQ and CLINC150 come from the release-gate reads. low, medium and high stop at layers 16, 20 and 32; auto uses the per-exit thresholds above. Images always run all 32 layers. Suite is the fifteen-benchmark mean without open_jev_ood.

³ v6.0-VL's final ECE as served by default, from its record (0.033 at high).

The gains at layer 32 are +0.023 on the suite (standard error 0.003), +0.033 on the held-out set (0.006) and +0.019 on MMLU-Pro. auto matches high at 19.5 layers on average: 52% of questions stop at layer 16, 35% at 20 and 13% go to 32. Calibration is worse than v6.0-VL's at every setting; it stays inside the release bar of 0.078 (§4.3). On CLINC150 the option-after-the-right-one rate is 0.7% (0.36% on the Decision Index's own CLINC150 rows, v6.0-VL 0.31%), inside the 5% bar; accuracy there is 0.837 against 0.843. The default (single questions at layer 32, the 0.95 cascade on multi-question requests) scores as high: run on every question, the bf16 package gives the same suite, held-out and MMLU-Pro figures as layer 32 at 30.5 layers on average. The low and medium ECE were not read at the served temperatures.

4.2 External: Decision Index

Decision Index 0.3, public index: 50.98. Full run of the default configuration (unset effort, τ 0.95), 140,178 requests, coverage 1.0: knowledge and reasoning 31.6, language 55.3, retrieval 57.1, tools 68.4, arts 45.7; 50.97 with the rows that also appear in our training data left out. Scored with the 0.3 kit before its release (pre-release kit). v6.0-VL scores 46.23 on the same index. On the public board of 2026-10-06 this ties the best 4B model (50.98 vs ezjev 4B s2's 50.82; the kit treats scores within 0.25 as tied) and is 27th of 113 overall. The board's official full score needs a submission, which is pending.

Decision Index 0.2.1, full run: 50.74 (all 150,759 requests answered, nothing cut, the unset effort; knowledge 32.5, language 55.5, retrieval 57.2, tools 68.3, arts 40.5), with the rows that also appear in our training data left out (50.75 with them). v6.0-VL: 46.24.

By effort level (Decision Index 0.2.1, the kit's 16k-question stratified sample, the same rows for every level and both models, one H200, bf16; overlapping rows removed; a sample read, not a full run):

v6.1-VL gains 4.6 to 5.7 points at every effort level. Unlike v6.0-VL, layer 32 now leads layer 20 on this sample (+0.66). auto reads 50.93, 0.5 below high, at 70% of its latency.

The second member's last mix was checked against the Decision Index kit before training, and full-item overlaps with its test rows were dropped. After the fact, RAGTruth and iSarcasmEval, the two benchmarks that moved most from v6.0-VL's zero, were checked for leaked kit items; none were found.

The single threshold. With no effort set, multi-question requests run the cascade with one threshold at both early exits. v6.0-VL's 0.59 does not carry over: at the refit temperatures the early exits are sharper, and 0.59 sends about 77% of questions to layer 16 (held-out set 0.718 in the served package). The single threshold was chosen again by auto's rule restricted to one shared value, written before the fit: 0.80 and 0.85 were feasible on half A, 0.85 won there, and on half B it went 0.08 layers over the 24-layer depth cap. The rule then falls back to 0.95, its most conservative member, which ships not confirmed. On half B it scores as layer 32 does (suite-like 0.785, held-out 0.745) at 26.8 layers. 0.80 passed on both halves; the rule does not pick it.

4.3 Calibration

Each exit has its own scalar temperature, refit on the average. Final-set ECE after calibration is 0.048 at layer 32 and 0.055 with auto, against v6.0-VL's 0.036 and 0.024; the release bar is 0.078. The early exits are further off. At temperatures fitted by the evaluation reader itself, not the served ones, final ECE is 0.086 at layer 16 against v6.0-VL's 0.025, and 0.058 at layer 20 against 0.043. If you route on usage.confidence at an early exit, check it on your own data.

5. Speed and serving

Median latency per request on the Decision Index sample, one NVIDIA H200, bf16:

v6.0-VL on the same setup: 40, 23, 27 and 40 ms for default, low, medium and high. Serving is unchanged from v6.0-VL: inputs up to 32,768 tokens, nothing cut (longer gets a 422); up to 5,120 options; own-token option pooling, set in meta.json.

The API returns a probability and a confidence, and they are different numbers: confidence is the peak statistic Jev documents, (K * p_max - 1) / (K - 1). Route on the probability of the option you intend to act on. The response also carries usage.depth, the layer each question was answered at.

6. Limitations

• Calibration is worse than v6.0-VL's: final ECE 0.048 against 0.036 at layer 32, 0.055 against 0.024 with auto, and further off at layers 16 and 20 (§4.3).
• The default single threshold, 0.95, is the fallback of the selection rule, not a confirmed choice (§4.2).
• Serve it as packaged. With the older whole-block option pooling, CLINC150 accuracy falls to 0.627 and the option after the right one is picked 23% of the time; at a 16k-token cap the long-input check cuts 446 rows and fails. The package sets own-token pooling and a 32,768-token cap.
• The development rows dropped before the temperature and threshold fits were checked against the last corpora of both members, not against every earlier corpus in either lineage. Rows those corpora trained on may remain in the development sets. v6.0-VL's selection used the same sets.
• The image rows of the development sets have no case id and could not be filtered. Images always run all 32 layers, so they cannot move a threshold, and they are not in the temperature fit.
• One seed per stage. Member B's head-stage seed was named before any averaged number was read; member B's own selection had picked the other seed.
• The early exits read text only; questions with images always run all 32 layers.

7. How it got here

25 arms ran after v6.0-VL's: one per distinct experiment id with a run record, first recorded after the run that chose v6.0-VL's auto thresholds and through the run that chose v6.1-VL's, a rerun counted once (24 without one analysis-only record). They span every line the loop ran in that window: data 8, calibration and exit policy 6, method 6, 27B 4, reinforcement learning 1. Two of them made this release; a third explains it. Member B was trained before the window and is counted with v6.0-VL's arms.

• Averaging two fine-tunes beat both. Pre-registered on half B of the development sets, the average led the better member by 0.0129 at layer 32 (z 2.08; 0.0184, z 4.4, with the rows a member trained on dropped). Read once on test, it led v6.0-VL by 0.023 on the suite and 0.033 on the held-out set.
• The gain is in the tower, not the heads. The averaged tower recovers 98% of the gain under v6.0-VL's own heads; averaging only the heads gives nothing. Uneven weights did not help.
• Averaging changes calibration. The averaged logits are smaller, so v6.0-VL's temperatures leave the model underconfident and its single threshold of 0.59 stops most questions at layer 16. The temperatures and both thresholds had to be chosen again.

The full list of arms on the released line is in EXPLORE.md.

Rerunning it

The code, the specs and the commands are in data/v6.1-vl_recipe/; D is the data directory and D/heads-s1 is v6.0-VL's stage-4 checkpoint. The training corpora of both members (v6.0-VL's text and image corpora, member B's three corpora) and the policy development sets are not public, so member B and the policy selection cannot be rerun from this repository alone. Serving the released model needs only the public code.

# member B, last two stages (earlier stages: data/v6.1-vl_recipe/0-lineage.json)
python scripts/release_train.py --model Qwen/Qwen3.5-4B-Base --seed 17 --root D \
  --spec data/v6.1-vl_recipe/1-member-b-domain-mix.json --corpus D/domain-mix-2 --save-dir D/b-domain-2/s17
RSIJEV_OPTION_POOL_OWN_TOKENS=1 python scripts/fit_head.py --ckpt D/b-domain-2/s17 --corpus D/domain-mix-2 \
  --dev-corpus D/domain-mix-1 --out D/b-heads-s1 --heads all --sdpa-math --max-length 4096 --skip-prefix vis --seed 1

# the average
python scripts/soup_checkpoints.py D/soup D/heads-s1 D/b-heads-s1

# per-exit dumps, rows a member trained on dropped, temperatures and thresholds, the package
python scripts/dump_exits.py dev --ckpt D/soup --corpus D/image-corpus --out D/dumps/soup   # and cases / vision / eval
python scripts/select_exit_policy.py overlap ...      # development case ids found in the members' corpora
python scripts/select_exit_policy.py exclude ...      # the dumps without them
python scripts/select_exit_policy.py thresholds ...            # refit temperatures; auto: 0.85 at 16, 0.50 at 20
python scripts/select_exit_policy.py thresholds ... --single   # one threshold: 0.95, the rule's fallback
python scripts/package_multiexit.py --ckpt D/soup --temperatures D/policy/auto.json \
  --single-tau D/policy/single.json --drop-exit 12 --thresholds D/policy/auto.json --out D/package

8. Corrections to v6.0-VL's record

None. The v6.0-VL record is left as published.