jevmlx

Typed decisions from a local LLM in one forward pass. Apple Silicon, MLX.

• You give a schema (Pydantic or JSON) and a context.
• jevmlx prefills once and decides every field in one batched pass: no text generation, no JSON repair.
• You get a validated typed object plus a per-field probability.

Pre-release. Accuracy/latency tables are generated by jevmlx eval and committed under benchmarks/results/ (first full run pending).

Inspired by and built on rorshopping/jev-on-a-laptop, the research repo that reproduced the technique on a laptop. jevmlx turns that study into an installable library. Unofficial, not affiliated with TypeSafe AI or Jev.

60-second demo

uv pip install git+https://github.com/bnsd55/jevmlx
jevmlx decide --preset fintech_fraud

Illustrative output:

Preset : FinTech Fraud & Autonomous AML Compliance (28 Fields)
Model  : mlx-community/Qwen2.5-1.5B-Instruct-4bit
Latency: 970.9 ms (prefill 416.6 + batched pass 530.4)

field                           value                   conf   type
------------------------------  ----------------------  -----  -----
is_fraudulent                   True                    0.825  boolean
risk_tier                       HIGH                    0.752  enum
recommended_action              BLOCK_TRANSACTION       0.702  enum
...

The same decision from Python:

from typing import Literal
from pydantic import BaseModel, Field
import jevmlx


class Fraud(BaseModel):
    is_fraudulent: bool = Field(description="Whether the transaction is fraudulent")
    risk_tier: Literal["LOW", "MEDIUM", "HIGH", "CRITICAL"] = Field(description="Risk tier")


context = "Wire transfer to a new IBAN, requested from a Tor exit node on an unrecognized device"

d = jevmlx.decide(Fraud, context, model="mlx-community/Qwen2.5-1.5B-Instruct-4bit")
d.value  # validated Fraud instance
d.confidence  # {"is_fraudulent": 0.xx, "risk_tier": 0.xx}

Why not just ask for JSON?

Generation can produce malformed or off-schema output and gives no per-field confidence. jevmlx reads probabilities directly from the model at the decision position. The whole object is assembled, never generated. One prefill plus one batched pass instead of N generated tokens.

How it works

[context + schema] ──► prefill (once) ──► KV cache
                                            │ broadcast ×N fields
      ┌───────────┬───────────┬─────────────┴───────────┐
    field 1     field 2     ...                      field N
      └───────────┴──── one batched forward pass ────────┘
                          │
        restricted next-token probs per branch → trie → pick value + P(choice)

1. Plan. The schema compiles into a batch plan: choice suffixes, shared prefixes, lead-ins (schema.py).
2. Prefill. The context plus a compact schema catalog goes through the model a single time.
3. Rows. Each field's choice continuations become trie rows against the broadcast KV cache; choices that share a first token get their own rows.
4. Batched pass. All rows are evaluated in one batched forward pass when they fit; otherwise the pass is chunked by a memory heuristic.
5. Trie scoring. At each branch point the model's next-token distribution is restricted to the allowed tokens, and P(choice) is the product of those branch probabilities; the JSON object is assembled from the winners (engine.py).

Two scoring modes: trie (default) and letter slots (--scoring letters). Letters list each field's choices as lettered options and read the next-token distribution at the field's slot position — no tokenization collisions by construction; try it when a schema's choices share long first tokens.

Features

1. One decision pass for every field. The KV cache is broadcast so all schema fields are scored in one batched suffix pass; latency grows with the longest suffix, not field count.
2. Always-valid JSON. The object is assembled from per-field decisions, never generated token by token, so it cannot be malformed; multi fields return the subset of options that apply.
3. Honest confidence. Each field's probabilities come from restricted branch distributions over a token trie and sum to 1 at full precision; jevmlx calibrate fits one temperature to correct overconfidence.
4. Typed Python API. Pass a Pydantic model, get a validated instance with per-field confidences — jevmlx.decide for one context, jevmlx.decide_many for many with one model load.
5. HTTP server. jevmlx serve --model M loads the model once and serves one decision per request on POST /decide (serial — one Metal GPU).
6. Temperature calibration. jevmlx calibrate fits one scalar temperature on labeled JSONL by minimizing NLL and reports binned ECE before and after.
7. Works with causal decoder models served by mlx-lm. Prompts use the tokenizer's own chat template — no hand-rolled role tags; tested models are in the table below.
8. Schema linting before you ship. jevmlx validate SCHEMA.json loads the tokenizer only (no model download) and reports first-token collisions with rename suggestions, duplicate choices, single-choice fields, and compile errors.

Model compatibility

Latency and memory numbers below come from benchmarks/compat.py. Accuracy and calibration numbers come from jevmlx eval / jevmlx report and live in benchmarks/results/.

Every preset field is decided in one batched pass — measured on a MacBook Pro M2 Pro, 34 GB, macOS (warm runs, benchmarks/compat.py). "Warm latency" is the average of the second run on both presets; peak memory is Metal's process high-water mark after both presets.

Evaluate and benchmark

Build the labeled JSONL, run it through a track, then summarize offline:

python -m benchmarks.to_jsonl --out cases.jsonl           # bundled fintech cases
python -m benchmarks.typesafe.fetch --out typesafe.jsonl  # TypeSafe public examples

jevmlx eval --data cases.jsonl --track parallel --out DIR
jevmlx report --predictions DIR/predictions.jsonl --out DIR/report.json

Three tracks: parallel (the constrained engine), naive_local (the same local model writing the JSON object itself, parsed strictly), api_baseline (an OpenAI-compatible chat model via --api-base / --api-model). Raw predictions, run manifests, and reports land in benchmarks/results/. To contribute results from your own Mac, see BENCHMARKING.md.

Roadmap

Where this is going next: ROADMAP.md. A correctness gate from an external review, an evaluation loop, latency profiling, then PyPI.

Contributing

Branch off main, run ruff check --fix . && ruff format . and pytest -m "not slow" before pushing. Details: CONTRIBUTING.md.

Credits & license

jevmlx is MIT-licensed (see LICENSE); third-party credits are listed in NOTICE.

• rorshopping/jev-on-a-laptop (MIT) — the research origin and inspiration; the study that reproduced the technique on a laptop.
• harshatheg/Qwen-2.5-1B-RLCD (Apache-2.0) — the original parallel constrained decoding engine whose approach jevmlx reimplements.
• TypeSafe AI's Jev — the product whose published technique this project reimplements; no affiliation, no access to their model.