RSI-Jev

A recursively self-improving research loop that builds Jev-style System One decision models.

🌐 shanghua-gao.github.io/RSI-Jev

🤗 Models · Docs · Every experiment · Release notes

AI agents propose the hypotheses, register their predictions before spending GPU time, run the experiments, and retire their own champions when the evidence says to. Every release, and every experiment that failed on the way, is published with its numbers. The loop is the next version of AutoScientists.

• 8 releases in 13 days, from v1.0 to v6.1-VL, each trained, evaluated and documented by the loop.
• 496 experiments, every one written up, failures included: EXPLORE.md.
• v6.1-VL 27B has the highest score on the Decision Index 0.3 public set: 65.64, with --effort auto. The official score is pending.
• v6.1-VL 4B scores 50.98 on the public Decision Index 0.3, up from 46.23 for v6.0-VL.

What it builds

Models that answer a yes/no, pick-one-of-k or rate-on-a-rubric question about a document, a chat or an image. One forward pass returns a calibrated probability for every option. Nothing is generated, so there are no reasoning tokens to spend.

What v6.1-VL can spend is depth. It has answer heads at layers 16, 20 and 32, and effort picks how many layers a request uses:

v6.1-VL 27B has the same levels at layers 48, 56 and 64.

Fast when it's easy, deep when it's hard. Which tasks need which depth: docs/effort-depth.md.

_{Capsules photo from VisA (Zou et al. 2022, CC BY 4.0, resized).}

Quickstart

pip install "rsi-jev[fast,vision] @ git+https://github.com/Shanghua-Gao/RSI-Jev"
rsi-jev serve v6.1-vl-4b --effort auto --port 8000

On a 96 GB GPU, rsi-jev serve v6.1-vl-27b --effort auto serves the 27B.

On a Mac with Apple silicon, the MLX builds of the 4B (8-bit, lossless; 4-bit, smaller) run with pip install "rsi-jev[mlx,vision] @ git+https://github.com/Shanghua-Gao/RSI-Jev" and rsi-jev serve v6.1-vl-4b-mlx-8bit --backend mlx: docs/mac.md.

curl localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "jev-latest",
  "state": "I was charged twice. Please refund.",
  "questions": {"refund": {"type": "noul", "instructions": "Does the user request a refund?"}}
}'

Or in Python, with no server:

from rsijev import Decider

d = Decider("v6.1-vl-4b")
d.decide("Customer photo: <image>\nThe customer says it arrived damaged.",
         {"damaged": {"type": "noul", "instructions": "Does the photo show visible damage?"}},
         images=["photo.jpg"])

The server speaks Jev's API, so an existing Jev client works once its base URL points here. Every response reports which layer answered and how confident it was. Options and limits: serve/README.md; setups and what each costs: docs/inference.md.

Releases

Decision Index is the public benchmark, full runs of the default configuration (the 27B with --effort auto, which we recommend): the 0.3 public index (public set; v6.1-VL 4B scored with the pre-release 0.3 kit) and 0.2.1. The held-out set is our zero-shot check: no release trained on it. The 4B and smaller are fine-tuned from Qwen3.5 Base; v6.0-VL and v6.1-VL 4B run the whole Qwen3.5-4B-Base and can answer at layer 16, 20 or 32 of it. v6.1-VL 27B is fine-tuned from Qwen3.8-27B and answers at layer 48, 56 or 64. Every number, caveat and per-benchmark score is in the release records.

• 2026-10-08 · v6.1-VL 27B brings the same method to Qwen3.8-27B. Decision Index 0.3 public set 65.64 with auto.
• 2026-10-07 · v6.1-VL 4B averages v6.0-VL with a second fine-tune of the same base. Decision Index 0.3 46.23 → 50.98.
• 2026-10-06 · v6.0-VL 4B chooses its depth per question. Decision Index 38.38 → 46.24.
• 2026-10-02 · v5.0-VL 3B cuts the model to the first 20 of 32 layers and says "unknown" when a question has no answer.
• 2026-10-01 · v4.0-VL reads images, with text held at v3.0's level.
• 2026-09-28 · v3.0 is the first release where reinforcement learning helps.

How it got here

_{The loop's first three releases: each grey dot is an experiment that did not beat the champion.}

The experiments that made a release are in docs/history.md; all the others, with why each failed, are in EXPLORE.md; the reinforcement-learning arms are in docs/rl.md.

Each experiment that moved a release, from v1.0 to v6.1-VL

where it went	result	what it established
v1.0	typed-decisions 0.662	the bar to clear, and it had never seen a benchmark's train split
train on the benchmark's own train split	typed-decisions 0.7794	worth +0.12 — and it costs general knowledge, so it was rejected
the same + 15% general-knowledge replay	typed-decisions 0.796	the replay pays that cost back. The first keeper
the other benchmarks' train splits too, 3k per source	suite 0.7049	the lever is not specific to one benchmark
a per-question confidence head, forbidden to sharpen score questions	suite ECE 0.0736	confidence that means something, and it never changes which option wins. Shipped
v2.0	suite 0.7056, ECE 0.0546	capability was v1.0's lever; what it trains on, and calibrating it afterwards, is a second one — and free at inference. But the gain was on the benchmarks it had trained for
the replay weighted toward science, half of it recast to ten options	MMLU-Pro 0.331	more and better-shaped data barely moves general knowledge: +0.008
freeze the tower's lower third instead	MMLU-Pro 0.339	holding those layers still does recover some of it — and breaks three of the decision benchmarks, because they do need to adapt
the bottom 8 layers at one tenth the learning rate	MMLU-Pro 0.359	it was never a data problem. Fitting decisions through the whole tower at one rate was overwriting what the knowledge sat in — frozen layers cannot adapt, slow ones can
v2.1	suite 0.7291, MMLU-Pro 0.383	and it travels: ahead of v1.0 on all three benchmarks it never trained on, where v2.0 was not
wider coverage (data-scale-2): +84k questions, emotion/SST-5/ANLI and more tasksource and Open-Jev train items	suite +0.011 / +0.013 on two seeds	coverage is still the lever that moves the suite
a two-layer MLP combining the option and cross-attention features	suite 0.7589 on the new 15-benchmark suite	the readout had a little left in it: +0.002
a listwise reranking reward (NDCG@5 over 16 candidates) on top, against a KL anchor	hippo R@1 0.192 → 0.308, suite −0.003	the first RL stage kept: a reward over the order of candidates carries what per-item labels cannot
v3.0	suite 0.756, ECE 0.066	the first release where RL works: +60% reranking R@1 over its supervised parent, the rest held level
image training on the train splits of twelve public datasets, with text replay	held-out image top-1 0.809 against 0.796 for a text-only control	the model reads images through the base model's frozen vision tower
generated reasoning questions with soft targets where the model is at chance	held-out ECE after calibration, BBH excluded, 0.096 → 0.075	the image rounds had made it overconfident; this repairs most of it
an RL stage charging a confident mistake 4x a timid correct answer	held-out ECE before calibration 0.200 → 0.082	against a matched supervised stage: better accuracy and calibration, no detectable ranking difference (one seed)
v4.0-VL	image top-1 0.803 held out, suite 0.756, ECE 0.043	reads images, with text held at v3.0's level
the same recipe on Qwen3.5-4B, read at layer 16, 20 or 24 of its 32	suite 0.744 · 0.760 · 0.760	a System One model doesn't need the deep layers: quality flattens by layer 20
images on the layer-20 model, plus questions whose right answer is "unknown"	KoBBQ unknown-when-ambiguous 0.679 → 0.891	images cost the model its "unknown" answer until the data asked for it
pool each option over its own tokens, not the separator that follows the previous one	CLINC150 0.383 → 0.753	a readout bug found by an outside benchmark; the same weights read correctly once each option is pooled over its own tokens
retrain only the decision head for that readout, tower frozen, 600 steps	final ECE after calibration 0.080 → 0.042, Decision Index 38.38	the head had learned the old pooling; a short head-only stage recovers the calibration
v5.0-VL	MMLU-Pro 0.429, Decision Index 38.38, KoBBQ unknown 0.932	3B: the first 20 of 32 layers
early-exit heads that read a detached copy of their layer, so no gradient from them reaches the trunk	MMLU-Pro 0.443 at layer 32	retuning heads on a trunk trained with attached exits had not recovered depth: the trunk had lost it
10,000 steps on a cleaned corpus instead of 40,000 on the old one	held-out +0.026, MMLU-Pro +0.043	longer training overfits the suite; shorter and cleaner wins held out
one temperature per exit and an exit policy fitted on held-out proxies of the test mix	every release bar passes at 20.9 layers on average	policies tuned on in-distribution rows stop too early on hard questions
v6.0-VL	Decision Index 46.24, MMLU-Pro 0.440	4B: answers at layer 16, 20 or 32, whichever is confident first
v6.0-VL averaged, weight 0.5 each, with a second fine-tune of the same base trained on other data	suite 0.793 and held-out 0.728 at layer 32, against 0.770 and 0.695	the average beats both members; the gain is in the tower, the heads add nothing
one temperature per exit and both thresholds chosen again for the average	auto at 0.85 / 0.50 matches layer 32 at 19.5 layers	averaging shrinks the logits: at v6.0-VL's temperatures it was underconfident; calibration ends worse than v6.0-VL's (final ECE 0.048 vs 0.036)
v6.1-VL	Decision Index 0.3 50.98, held-out 0.729	4B: v6.0-VL averaged with a second fine-tune
two LoRA fine-tunes of Qwen3.8-27B, averaged weight 0.5 each, chosen from 9 full Decision Index 0.3 reads	Decision Index 0.3 65.59, against 64.83 and 65.30 for the two members	averaging carries over to 27B; uneven weights and a third ingredient read 65.37 to 65.58
short inputs right-padded to a multiple of 64 tokens and replayed as CUDA graphs	short one-question requests in about half the time	at one request a time the 27B is bound by kernel launches, not compute
v6.1-VL 27B	Decision Index 0.3 public set 65.64 with auto, held-out 0.808	27B: the same method on Qwen3.8-27B, answers at layer 48, 56 or 64

Documentation

Contributing

A case where the model is confidently wrong, or a variant you tried that failed, is the most useful thing you can send: report a wrong answer · report an experiment. Each becomes a registered prediction tested in the next version. See CONTRIBUTING.md. To collaborate or support the work with compute, contact Shanghua Gao.

Acknowledgements

Thanks to HP and NVIDIA for providing the HP ZGX Nano AI Station, powered by the NVIDIA GB10 Grace Blackwell Superchip.

License

Code: MIT. Weights: Apache-2.0, following the base model; some image training sources are non-commercial, listed on each model card. Not affiliated with TypeSafe AI. To cite, use CITATION.cff.