Jeff-Gemma4-E2B

The Jeff models are fine-tunes of Qwen3.5 and Gemma 4 for zero-shot classification: small, fast decision models you slot into your code. You describe a situation and list the options in plain words; Jeff returns a calibrated probability for each option from a single forward pass. No generated text, no parsing: this model takes about 29 ms per decision on an RTX PRO 6000 (the Mac's MLX backend runs the Qwen models only).

Zero-shot means the options can be anything: support queues, user intents, moderation labels, voice commands, game moves. Your categories don't need to appear in the training data; you describe them, and Jeff picks.

What it is, and what it isn't. These are very small models. They make extremely fast, well-calibrated judgement calls between options, and they slot easily into your local code. On benchmarks they approach, and sometimes beat, Jev; but at this size their reasoning won't match Jev's, which runs on a much larger model. If zero-shot accuracy isn't good enough for your purposes, a short fine-tune on your own examples takes you much further: our voice-navigation fine-tune moved held-out accuracy from 31.7% to 95.8% in under half an hour on one GPU.

Built entirely on local hardware. Training on one RTX PRO 6000 workstation GPU (the 0.8B trains in about 2 hours, the 2B in about 3.5), all synthetic training data written by an open model (Qwen3.8-Flash-Next) on two DGX Sparks, testing on a MacBook. No cloud GPUs, and no closed-model output in the training data; a closed model was used only to spot-check the quality of a sample of the synthetic data.

Independent project. Jeff uses the same request format as Jev, but it is not affiliated with or endorsed by TypeSafe, the makers of Jev. Our training code starts from the open-source AutoJev recipe.

What it does

{
  "model": "jeff-latest",
  "state": {"voice_transcript": "open the engagment leter", "current_screen": "Deal overview"},
  "questions": {
    "intent": {
      "type": "choice",
      "instructions": "Which of these does the user want?",
      "criteria": {"1": "Engagement letter", "2": "Inbox", "3": "Deal settings"}
    }
  }
}

The answer is a probability per option ({"1": 0.94, "2": 0.03, "3": 0.03}), the chosen option and a confidence. Three question types: choice (pick one of up to 255 options), noul (yes/no, returned as a probability) and score (a point on a scale you describe). Several independent questions in one request are answered together.

Benchmarks

4,599 questions from five public benchmarks, plus JevBench's public hard tier (105 items, scored separately):

Bold: the winner of Jeff against Jev in each row: a Jeff score above Jev's published figure, or Jev's figure where it beats every Jeff model. Bold italic: AutoJev-27B where it is the best of all models in the row (on RAGTruth, tied with Jeff-Qwen3.5-2B); it is shown for reference, since the head-to-head comparison is with Jev. The published Jev and AutoJev figures were measured on a different sample of the same benchmarks. Jeff's overall score comes from classification and grounding (Financial PhraseBank, RAGTruth), where it matches or beats the large models; on the reasoning-heavy benchmarks (BBH, JudgeBench, JevBench) it stays well below them, as you would expect at this size.

Games: a zero-shot test

To test zero-shot performance on tasks unlike anything in the benchmarks, we had Jeff play three games. Games aren't the ideal zero-shot test, since a game's state isn't typical unstructured data; but they are a common, and fun, way to test a System 1 model. Each turn, the code describes the situation and the legal moves in words, and the model picks one. The options state what each move leads to (Frogger: "you would be hit by a car and lose a life"; Doom: "the nearest monster is a little to your left"), but never which move is right. Each result is 20 episodes, seed 1234; ▶ opens a video of the run's first episode.

Jeff-Qwen3.5-0.8B playing, zero-shot (the bold row in the table below):

Jeff-0.8B decides in 29–49 ms per move on an M4 Max; Jev's published Doom run took 212 ms per call over its API. The two times were not measured on the same hardware.

Speed and size

Median time per decision over the same 200 benchmark questions (about 200 input tokens each), one question at a time, from raw text to probabilities:

How to use

git clone https://github.com/firelex/jeff && cd jeff
uv sync
uv run hf download mstrasser/Jeff-Gemma4-E2B --local-dir Jeff-Gemma4-E2B
JEFF_CHECKPOINT=Jeff-Gemma4-E2B PORT=8765 uv run jeff-serve                        # NVIDIA or CPU
curl -s localhost:8765/v1/systemone -H 'content-type: application/json' -d @request.json

• Reason in code, decide with Jeff. State each option's consequence; don't ask Jeff to forecast.
• Use short option keys and descriptive text: {"1": "Engagement letter"}, not long IDs, which cost time and add nothing.
• Fine-tune for your domain. A voice-navigation fine-tune on ~11k app-specific examples took held-out accuracy from 31.7% to 95.8% in one epoch.

Training

• Recipe: full-weight supervised fine-tuning, one epoch (learning rate 5e-6 for the 0.8B, 1e-5 for the 2B), batches of 256, cross-entropy over the option letters, then one scalar temperature fitted for calibration. Checkpoints are chosen on a held-out development set; the benchmark panel is never used for selection.
• Data (271k questions): public datasets converted to decisions (entailment, QA, sentiment, safety, fact verification and more), the training splits of WinoGrande (40k) and RAGTruth (15k), long real documents (ContractNLI, CUAD, ConditionalQA), 10k code-built probability questions with exact answers, ~31k synthetic questions written and checked by a local teacher model (Qwen3.8-Flash-Next on DGX Sparks), hijack attempts planted in 3% of questions, and "none of these" options added to 5%.
• Leak filter: every training question is checked against the benchmark panel and JevBench; 50 near-duplicates were removed.
• Disclosure: at least half of each training family is written in the benchmark panel's layout conventions (how states, questions and options are formatted). No panel item is used in training, but matching the format helps the score.
• Hardware: one RTX PRO 6000 (96 GB) for training, DGX Sparks for the teacher, and an M4 Max for local tests.

Caveats

• Small models don't reason. Expect fast, calibrated choices between the options you describe, not multi-step reasoning. At 0.8B–2B parameters this holds for every model, not just Jeff.
• Jeff-2B is a weaker game player than Jeff-0.8B. The untrained 2B already appears more risk-averse than the untrained 0.8B (in Doom it prefers turning away from the nearest monster; in Pac-Man it survives much longer but hesitates), and our training seems to have made that worse: Jeff-2B reverses direction in Pac-Man 3.5× as often as the untrained 2B. This needs more investigation.
• Benchmark scores don't predict game play. The untrained Gemma 4 E2B beats the untrained Qwen models on the benchmarks (62.5% against 45–47%) yet plays the games worst: it makes the right move most of the time but not reliably, and in a real-time loop the occasional wrong call compounds. Training fixed its Pac-Man (3.2 → 53.2 pellets) but not its Doom or Frogger.
• Prompts matter. The game results depend on options that state consequences in words. Jev's own Doom prompt (a raw bearing number plus an aiming rule) does not work for any of our models, trained or untrained.
• Different samples. The published Jev and AutoJev numbers were measured on a different sample of the benchmarks.
• English and text only. Calibration is fitted on our development data; recalibrate for a very different domain.

Licence and data

Weights: Apache 2.0 (from Gemma 4 E2B by Google; see Google's Gemma 4 licence). The training data mixes datasets under various licences, listed with their sources in docs/data-sources.md. We release the weights and code, not the training data; some sources are share-alike (CC BY-SA).