jevify-qwen3.5-4b-t2

Newer on the same backbone: jevify-qwen3.5-4b-readout-coh — readout fine-tuning with a coherence penalty, no heads: 0.751 accuracy, ECE 0.058, TVD to human labels 0.303 (FINDINGS 18). This repo stays the reference for its tier.

jev-bench · leaderboard · findings · code

A System One decision model: it does not write text. It reads a state, answers typed questions, and returns calibrated probability distributions your code can branch on.

This repo holds only what Jevify adds to Qwen/Qwen3.5-4B: 2,891,275 parameters of trained decision heads (11.6 MB) and a rank-16 LoRA adapter (85 MB, 21,233,664 parameters) that is merged into the backbone at load plus the calibration recipe. The backbone is pulled from its own repo at load time, so nothing is duplicated or relicensed.

Results on jev-bench

Scored on all 22,773 test records, against TypeSafe's Jev 1.13.0 on the identical records.

Across the whole benchmark: macro accuracy 0.747 against Jev's 0.733, ECE 0.110 against 0.113, and 0.347 mean distance to human label distributions against Jev's 0.432 — lower is better, and that last number is the one a calibration claim rests on.

Use it

pip install git+https://github.com/uspraveen/Jevify

from jevify import load_jevified

model = load_jevified("Praveenrajus/jevify-qwen3.5-4b-t2")
answer = model.ask(
    state={"ticket": "I was charged twice for order A-104, please refund the duplicate."},
    questions={
        "dept": {"type": "choice", "instructions": "Which team should handle `ticket`?",
                 "criteria": {"billing": "Payments and refunds", "shipping": "Delivery problems", "other": None}},
        "refund": {"type": "noul", "instructions": "Does `ticket` ask for a refund?"},
        "anger": {"type": "score", "instructions": "How angry is the customer?",
                  "criteria": ["calm", "annoyed", "furious"]},
    },
)
print(answer["dept"]["choice"], answer["dept"]["confidence"])
print(answer["refund"]["noul"])

As a drop-in for the TypeSafe API

jevify-serve --model Praveenrajus/jevify-qwen3.5-4b-t2 --port 8000

TYPESAFE_BASE_URL=http://localhost:8000 python your_existing_typesafe_code.py

The official typesafe-sdk works against this unchanged — that is a test in the repo.

How it was built

Decision heads read the backbone's hidden state at each option's own line, so they score what an option means rather than how likely its identifier token is. They are applied as a residual on the model's own log-score — score_i = w·lm_i + f(...) with f zero-initialized — so training starts exactly at the untrained baseline and can only add to it. Trained on 5,885 records from 16 sources with at most 16 options each, which is what keeps the head usable at any K.

This is a Tier 2 model: a rank-16 LoRA on the backbone's attention and MLP projections was trained jointly with the heads (LoRA learning rate 0.0001, 3 epochs, best epoch 1 by validation loss), so the backbone can learn to put the judgment at the option's position instead of only being read there. The adapter is merged into the weights when the model loads, so inference costs exactly what the plain backbone costs.

Training optimizes strictly proper scoring rules directly — log score for Choice and Noul, ranked probability score for the ordinal Score — so calibration is the objective rather than a post-hoc repair. No reinforcement learning is involved: with a differentiable head the calibration objective is just a loss.

Six sources were held out of training entirely so generalization to unseen question types is measured rather than assumed. Full method, findings and limitations: github.com/uspraveen/Jevify · FINDINGS.md

Limitations

• English-first, text only, following the benchmark it was tuned on.
• Ordinal (score) questions on scales unlike those in training are the weakest case.
• The heads are trained on jev-bench's own train splits, so "held out" means held-out source, not a wholly different data universe.
• This run is one seed. The seed study in FINDINGS §6.3a found held-out accuracy to move by ±0.05 across seeds for a 2B residual head, so differences of that size between models are noise, not signal.