autotrust/JEV-27B-VL

JEV-27B that can see: the same System 1 decisions and System 2, now on images

autotrust/JEV-27B-VL is autotrust/JEV-27B with vision.

New (3 October 2026): robot arm and computer use

Every step below is one System 1 decision: a camera image or a screenshot in, a probability for every action out, in a single forward pass.

Robot arm: pick and place from a camera image. At every step System 1 looks at the top camera image and answers two questions: is the target left or right of the gripper, and above or below it? The arm moves accordingly and halves its step whenever an answer flips. It grasps the cube, carries it and drops it in the tray (MuJoCo simulation).

75% of 20 random scenes completed, the best of the JEV family. Every cube it grasped ended in the tray (15 of 15); every miss was a grasp 3.0–3.7 cm off target. About 240 ms per decision.

Computer use: screenshot → which element to click. A real browser (headless Chromium). Every clickable element gets a numbered box; System 1 picks the next click (or "the task is complete"), the browser clicks it, and the loop repeats.

95% of 60 random multi-step tasks completed (shop, settings, mail; 3–7 clicks each), about 0.26 s per click. The colour swatches carry no text, so that click is decided from the screenshot alone.

Same scenes and tasks for every model in the family:

• Use System 1 for simple visual questions inside a control loop. Asked to pick one of 8 motor commands directly, it completed 0 of 10 scenes.
• For computer use, give it the element text (as an accessibility tree would). With the numbered boxes alone it often declares the task complete too early.

Demo code: JEV-9B vl/demos/ (set JEV_URL to this server). Per-episode results: reports/demos/.

New (1 October 2026)

• System 1 over plain HTTP. serve_decide.py adds POST /v1/decide to the vLLM server: send {kind, state, question, options} and get a calibrated probability for every option, for text or images. See Quick start.
• Up to 256 options per choice question, with no retraining. Zero-shot on CLINC150 with all 150 intents as options: 89.5% with intent names alone, 93.8% with a one-line description per option. See Choice questions with up to 256 options.
• How to write prompts, measured. One-line "use when" descriptions for similar options help; question wording, JSON vs plain text and extra instructions make no measurable difference. See Writing System 1 prompts.
• 256K-token prompts. Decisions that hinge on one sentence hidden at a random depth in up to 250K tokens of text: 20 of 20 correct at every length tested. See Context length.

Highlight: zero-shot short-video recommendation (for TikTok-style feeds)

A short-video feed has to decide, for every user and every new clip, whether to show it. JEV-27B-VL System 1 can make that call from the pictures alone: it looks at the covers of the last 5 videos a user watched plus one candidate cover, and returns P(this user watches it) in one forward pass. It needs no interaction logs, no item embeddings and no training, so a brand-new video can be ranked the moment it is uploaded, before anyone has watched it.

Offline evaluation on MicroLens-100k, a public dataset from a real short-video platform with the original covers: for 200 users, the video each one actually watched next is hidden among 19 videos other users watched within ±3 days (what the feed would have been showing), and every method ranks the 20 candidates.

• Matches collaborative filtering with zero behaviour data: the same AUC as item-based CF learned from 59,045 users' watch histories (difference 0.000, 95% interval −0.041 to +0.040), and a higher top-5 hit rate (59% vs 49%).
• Solves the cold-start problem: collaborative filtering needs co-watch history; JEV only needs the cover, so new videos and new creators can be recommended from the first second.
• Pictures beat words: covers alone score +0.078 AUC over titles (95% interval +0.031 to +0.126).
• Fast enough for a ranking stage: 20 candidate covers scored in about 2.6 s on one GPU in the live demo.

Code, data pipeline and a live web demo: JEV-27B-DEMO / 07-video-recommendation.

Watch it decide

Every move below is one System 1 decision: a probability for each possible action, read off in a single forward pass. The panels show the action probabilities and the time each decision took.

Super Mario Bros. Each step is a movement choice plus a jump decision, about 150 ms per decision (video at 5× speed).

Rubik's Cube. Solving a 25-turn scramble, choosing each move from 11 standard formulas, about 46 ms per decision (10× replay).

Tetris. Choosing the column and orientation of each piece, about 122 ms per decision (5× speed).

Quick, Draw! Guessing what a player is sketching, stroke by stroke, among 16 answers. On 320 real players' sketches from the Quick, Draw! dataset it names the drawing 88% of the time once it is finished, and 62% of the time when only 60% of the strokes are down (random guessing: 6%). About 270 ms per guess.

Snake. Looking at the board and a list of positions, choosing a direction at every step: 18 pieces of food in 131 steps, about 300 ms per decision (4× playback).

Highlight: agent judge

Plan-RewardBench (ACL 2026). Which of two complete trajectories of a tool-using agent is better? On 1,171 pairs (planning, error recovery, safe refusal, tool irrelevance), System 1 reaches 73.2% macro-average accuracy, the top of the paper's table (Table 4): Qwen-Plus 70.0, DeepSeek-V3.2 69.6, Inf-ORM-Llama3.1-70B 69.2, Gemini-3-Flash 69.1, GPT-5 68.5.

AgentRewardBench (2025, with screenshots).

Did the web agent really finish the task? Given the user's goal, the agent's actions and final message, and the final screenshot, System 1 returns P(task completed) in one forward pass, zero-shot. On AgentRewardBench (McGill, 2025; 1,302 expert-labelled trajectories from GPT-4o, Claude 3.7, Llama 3.3 and Qwen2.5-VL agents) its precision is higher than every judge on the official leaderboard at that judge's own recall:

At threshold 0.5: precision 78.4, recall 70.2; AUROC 0.91. All 1,302 judgements in about 4 minutes on one GPU. Code: JEV-27B-DEMO / 10-agent-judge.

Highlight: multimodal judge

Given an image, a question and two answers, System 1 says which answer is better, zero-shot and in one forward pass per order. On VL-RewardBench (CVPR 2025, 1,247 human-verified pairs) it reaches 78.3% overall accuracy, above every model on its official leaderboard (26 models, updated May 2025):

2,494 decisions in 163 seconds on one GPU. Code: JEV-27B-DEMO / 09-multimodal-judge.

On the newer Multimodal RewardBench 2 (Meta, December 2025), which includes the current generation of judges (1,000 expert-annotated pairs per task; other rows from the paper's Table 2):

Text-to-image judging comes within 1.3 points of GPT-5; the four-task average is at the level of GPT-4.1 and Qwen3-VL-32B. GPT-5 and Gemini 3 Pro stay ahead, and image editing is the weakest task.

Benchmarks

JEV-27B-VL gives the same text decisions as autotrust/JEV-27B: on 1,000 answer-checking decisions the mean probability difference is 0.010 and 99.8% fall on the same side of 0.5. The text results below were measured with JEV-27B; the image result was measured with JEV-27B-VL.

Six public decision benchmarks

Scores in %, higher is better. JEV-27B and TypeSafe Jev 1.13 (hosted API) were run in full by AutoTrust (27 September 2026); the other rows are as reported in the NeoHorse-Jev-4B evaluation.

System 1 fidelity and calibration

Held-out test_set_30k of jev-distill-corpus-v3.

Independent benchmark: decision-models-under-pressure

Human gold labels (CLINC-150, MTOP, GoEmotions, DBpedia), 800 items.

Applied tasks (from JEV-27B-DEMO)

System 2

Quick start

hf download autotrust/JEV-27B-VL --local-dir JEV-27B-VL
bash JEV-27B-VL/serve.sh          # vLLM on :8000; one GPU with 80 GB or more

serve.sh runs serve_decide.py: the standard vLLM OpenAI server, with the same flags as vllm serve, plus a POST /v1/decide route for System 1.

python3 JEV-27B-VL/serve_decide.py --model JEV-27B-VL --served-model-name autotrust/JEV-27B-VL \
  --enable-lora --max-lora-rank 32 --lora-modules jev-decision=JEV-27B-VL/adapter_vllm \
  --logprobs-mode processed_logprobs --max-model-len 32768 --enable-prefix-caching --mamba-cache-mode align \
  --limit-mm-per-prompt '{"image": 8}' --max-num-seqs 8 --trust-request-chat-template

Set MAX_MODEL_LEN=262144 before serve.sh for the full 256K context (see Context length).

System 1: POST /v1/decide

Send a question and get back a calibrated probability for every option. The server builds the decision prompt, reads the option tokens and applies the decision head's bias and temperature, so any HTTP client works.

curl localhost:8000/v1/decide -H 'Content-Type: application/json' -d '{
  "kind": "choice",
  "state": "Customer: my card was charged twice for one coffee.",
  "question": "Which team should handle this?",
  "options": ["billing", "shipping", "tech support"]}'

{"kind": "choice", "effective_kind": "choice", "options": ["billing", "shipping", "tech support"],
 "probabilities": [0.9969, 0.0000, 0.0031], "choice_index": 0, "choice": "billing",
 "adaptation": "native", "protocol": "jev27-bare-v1", "model": "autotrust/JEV-27B-VL",
 "usage": {"prompt_tokens": 49, "completion_tokens": 1, "total_tokens": 50}, "num_model_requests": 1, "elapsed_seconds": ...}

For images, make state a list; each image is placed where it appears. Images can be data URLs or https:// URLs.

import base64, requests

def image(path):
    return {"image": "data:image/jpeg;base64," + base64.b64encode(open(path, "rb").read()).decode()}

def decide(kind, state, question, options=None):
    body = {"kind": kind, "state": state, "question": question, **({"options": options} if options else {})}
    r = requests.post("http://localhost:8000/v1/decide", json=body).json()
    return dict(zip(r["options"], r["probabilities"]))

decide("noul", ["Photo: ", image("photo.jpg")], "Is this scenario one where: the image shows food or cooking?")
# e.g. {'false': 0.00, 'true': 1.00}
decide("choice", ["Listing photo: ", image("item.jpg"), "\nSeller title: wireless earbuds, barely used"],
       "Which category fits this listing?", ["electronics", "clothing", "home and kitchen", "toys"])

Text-only decisions give the same results as autotrust/JEV-27B. GET /v1/decide/info reports the option limit and the temperatures.

import json, math, requests

B = "JEV-27B-VL"
DH = json.load(open(f"{B}/adapter_vllm/decision_head.json"))
T = json.load(open(f"{B}/calibration.json"))["per_kind"]
RAW = ("{%- for m in messages -%}{%- for c in m['content'] -%}{%- if c['type'] == 'text' -%}{{ c['text'] }}"
       "{%- else -%}<|vision_start|><|image_pad|><|vision_end|>{%- endif -%}{%- endfor -%}{%- endfor -%}")

def decide_mm(kind, state_parts, question, options=None):
    """state_parts: list of str and {'image_url': {'url': ...}, 'type': 'image_url'} parts. Up to 16 options."""
    options = {"noul": ["false", "true"], "score": [str(i) for i in range(6)]}.get(kind, options)
    lines = options if kind != "choice" else [f"{'ABCDEFGHIJKLMNOP'[i]}) {o}" for i, o in enumerate(options)]
    content = [{"type": "text", "text": f"[kind] {kind}\n[state] "}]
    content += [p if isinstance(p, dict) else {"type": "text", "text": p} for p in state_parts]
    content.append({"type": "text", "text": f"\n[question] {question}\n[options]\n" + "\n".join(lines) + "\n[decision]:"})
    s = DH["slots"]["ranges"][kind][0]
    ids = DH["verbalizer_ids"][s: s + len(options)]
    r = requests.post("http://localhost:8000/v1/chat/completions", json={
        "model": "jev-decision", "messages": [{"role": "user", "content": content}], "chat_template": RAW,
        "add_generation_prompt": False, "add_special_tokens": False, "max_tokens": 1, "temperature": 1.0,
        "top_k": 0, "top_p": 1.0, "logprobs": True, "top_logprobs": len(options), "allowed_token_ids": ids,
        "return_tokens_as_token_ids": True}).json()
    lp = {int(t["token"].split(":")[1]): t["logprob"] for t in r["choices"][0]["logprobs"]["content"][0]["top_logprobs"]}
    z = [(lp.get(t, -1e9) + DH["bias"][s + i]) / T[kind] for i, t in enumerate(ids)]
    e = [math.exp(x - max(z)) for x in z]
    return {o: x / sum(e) for o, x in zip(options, e)}

System 2 on images

chart = {"type": "image_url", "image_url": {"url": image("chart.png")["image"]}}   # OpenAI-style image part
requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "autotrust/JEV-27B-VL",
    "messages": [{"role": "user", "content": [chart, {"type": "text", "text": "What does this chart show?"}]}],
    "max_tokens": 1024, "chat_template_kwargs": {"enable_thinking": False}})

Choice questions with up to 256 options

choice takes 2–256 options in one question. The first 16 are labelled A–P, the labels the decision head was trained on. Beyond 16, the labels continue with Q–Z and then two-letter labels that are single tokens (AA, AB, …), read the same way, with no retraining. The response's adaptation field says native (up to 16 options) or wide-labels.

Zero-shot intent classification with every intent offered at once (accuracy on 400 test utterances per row):

• One choice question beats one yes/no question per option, and costs one forward pass instead of one per option: 83.2% vs 78.0% on MASSIVE, 78.8% vs 67.0% on BANKING77 (64 options), 94.5% vs 82.0% on CLINC150 (64 options).
• Calibration at 150 options (CLINC150): ECE 0.079 with names only (mean confidence 0.84 against 89.5% accuracy, slightly under-confident) and 0.034 with descriptions.
• Speed (one B200, 8 requests in flight, --max-num-seqs 8): a 150-option question takes a median 1.4 s with names only (about 790 prompt tokens) and 2.6 s with descriptions (about 3,100 tokens). Above 128 options the server reads the option tokens in two passes; the second reuses the cached prompt.

Writing System 1 prompts

What we measured on the intent-classification sets above (400 utterances per variant):

1. Facts in state, one question in question. state holds everything the decision depends on; question asks one thing about it. Plain text and JSON both work: a JSON state scored the same as plain text (83.8% vs 83.2%, 79.0% vs 79.0%, 94.5% vs 94.8%).
2. Ask about all candidates in one choice question, not one yes/no question per candidate (see above).
3. Give similar options a one-line "use when" description. This was the only change that clearly helped: +4.8 points on BANKING77 (64 options), +2.5 on CLINC150 (64) and +4.3 on CLINC150 (all 150). The line should say what separates the option from its neighbours:

   [options]
   A) card arrival: use when users ask about the status or delivery of a physical card.
   B) card delivery estimate: use when users ask about the expected arrival time or delivery speed of a new card.
   C) Refund not showing up: use when a customer cannot see a completed refund in their account or statement.
   D) request refund: use when a user asks how to initiate or the timing of a refund for a purchase.
   

   More text did not help further. Longer rules with "do not use for …" clauses (about three times the tokens) and two worked examples per option scored 83.8% and 84.0% on BANKING77 (vs 83.8% for one line), and 97.3% and 96.0% on CLINC150 (vs 97.3%). Generic descriptions did not help either: CLINC150's own intent descriptions scored 94.5%, against 94.8% for names alone.
4. Wording barely matters. A domain-specific question, an extra instruction ("Choose the single best-matching intent"), a JSON state and alphabetical option order all stayed within ±1.5 points of the plain prompt, inside the ±1.8-point sampling noise of 400 items. Keep prompts plain.
5. Phrase noul so that "true" is the outcome you want the probability of, e.g. "Is the customer asking for a refund?". Our demos use "Is this scenario one where: …?" for properties of a text or an image.
6. One option per line. The template lists one option per line, so keep newlines out of option text.
7. Check order sensitivity when it matters. About 7% of 16-option answers change with option order alone. For pairwise judging, ask both orders and average; on RewardBench pairs the two orders agreed 96% of the time.
8. Use the probability, not just the top option. Act automatically above a threshold you validated on your own data, and send the rest to System 2 or a person.

Context length

The backbone's native context is 262,144 tokens (256K), and System 1 accepts prompts up to that length when vLLM is started with --max-model-len 262144. We tested decisions that hinge on a single sentence placed at a random depth in long real text (concatenated PubMedQA abstracts): a yes/no question (was the refund for order #N approved or rejected?) and a 16-option question (which room is the spare part for machine M kept in?), 10 of each per length.

Latency is one request at a time on one B200 and is almost all prompt processing (about 8,700 tokens per second at 250K).

Memory: the KV cache takes about 65 KB per token, so a full 256K prompt needs about 17 GB on top of the 52 GB of weights (tested on a 183 GB B200). On an 80 GB GPU, start with --max-model-len 131072.

Serving notes

• --max-num-seqs 8 is required. With more than 8 sequences in one batch, vLLM's LoRA path for this multimodal model class returns wrong System 1 probabilities (the text-only JEV-27B is not affected). With the cap, results match JEV-27B at any client concurrency; requests beyond 8 simply queue.
• Throughput on one B200 with the cap: 24 short text decisions in about 1.4 s; 4,000 image decisions with 6 images each in 316 s (about 13 per second).
• --trust-request-chat-template lets /v1/decide (and the client-side code above) render the raw decision template for image decisions. System 2 uses the model's own chat template.
• serve_decide.py needs the vLLM development build this model was tested with (September 2026): it uses logprob_token_ids and that build's server layout. Everything else is plain vLLM, and all OpenAI endpoints stay available on the same port.
• Calling /v1/completions or /v1/chat/completions for System 1 yourself: pass top_k: 0 and top_p: 1.0. The model's generation config sets top_k=20 and top_p=0.95; vLLM applies them as request defaults, and they would truncate the returned probabilities. serve_decide.py does this for you.
• Images are resized by the Qwen3.8 processor. Downscaling large images first (for example to at most 448 px) keeps the number of vision tokens and the latency low.

Limitations

• System 1's decision head was trained on text. Decisions over images are zero-shot: they are well ordered in the experiment above, but their calibration on image tasks has not been measured systematically.
• The short-video recommendation result is an offline evaluation on one dataset (200 users), ranking from covers only.
• Choices with more than 16 options use labels the decision head never saw in training. Accuracy holds up (see the intent-classification table), but their calibration has been measured on one dataset only.
• The long-context test is a single-fact decision over text. Long documents with images and harder long-document reasoning have not been measured.

License

Apache-2.0. This repository contains the weights of Qwen/Qwen3.8-27B (Apache-2.0, see LICENSE) unchanged, plus the JEV System 1 adapter and decision head from autotrust/JEV-27B.