train-jev-4b-lora

A JEV-style System 1 decision model: Qwen3.5-4B + LoRA + a 24-slot decision head, trained with train-jev. It does not generate text. It returns a calibrated probability for every option in one forward pass, so you can act on confident answers and route the rest to a person. Yes/no, multiple-choice and rating questions; the server speaks TypeSafe's System One API.

The small, fast one. Ahead of Kev-4B at the same size (13-dataset PK accuracy 0.647 vs 0.640, Decision Index 40.4 vs 38.4) with the lowest PK calibration error of all models compared (ECE 0.076). Runs on any 24 GB GPU, 43 ms per request. For the highest accuracy use train-jev-27b-lora.

Quick start

git clone https://github.com/shibing624/train-jev.git && cd train-jev
pip install -r requirements.txt
python -m trainjev.serve --base Qwen/Qwen3.5-4B --adapter shibing624/train-jev-4b-lora --port 8020

curl -s localhost:8020/v1/systemone -H 'content-type: application/json' -d '{
  "state": "Shoes arrived two weeks late and in the wrong size. Also I see two charges on my card.",
  "questions": {
    "department": {"type": "choice", "instructions": "Which team should handle this?",
                   "criteria": {"returns": null, "shipping": null, "billing": null}},
    "escalate":   {"type": "noul", "instructions": "Does this need urgent human attention?"},
    "frustration": {"type": "score", "instructions": "How frustrated is the customer?",
                    "criteria": ["Calm", "Frustrated", "Very angry"]}
  }}'

This repo holds the LoRA adapter, the decision head (head.pt) and the temperatures (calibration.json); the backbone is downloaded on first start. Load it through train-jev: plain PEFT / Transformers loading applies the adapter but not the head or the temperatures. More: serving guide.

Results

Same questions for every model. ↑ higher is better, ↓ lower is better.

Full protocol and breakdowns: docs/results.md.

Training

Qwen3.5-4B with a rank-16 LoRA on every attention, DeltaNet and MLP projection, and a 24-slot fp32 head (false/true, 0–5, A–P) initialised from the backbone's lm_head. Trained for 5,123 steps (batch 128) on SargeDev/jev-distill-corpus-v3 with KL to the teacher distribution plus 0.5 × RPS on rating questions; per-type temperatures fitted on a held-out calibration split. Recipe: docs/training.md.

Limitations

• Calibrated in distribution; check a probability threshold on your own data before relying on it.
• Only the first 16 choice labels (A–P) are trained; choices with more than 16 options use extra, untrained labels.
• Trained for decisions only, on a single seed.

Citation

@misc{train-jev,
  title={train-jev: Training JEV-style Decision Models},
  author={Ming Xu},
  year={2026},
  howpublished={\url{https://github.com/shibing624/train-jev}}
}

Independent research reproduction, not affiliated with TypeSafe AI or autotrust. Apache-2.0.