OpenJev

Fast, calibrated, typed decisions from an open model. OpenJev is an open-source "System One" decision server. Send it a state and a set of typed questions (yes/no, choice, score). It returns a probability and a confidence for every answer in tens of milliseconds. It reads the answers straight off the model's probabilities. It generates no text and parses nothing, so an answer cannot go off-schema. Questions can also ask about images.

OpenJev speaks the same wire API as TypeSafe's Jev, so their SDKs work against it unchanged. It runs DiffusionGemma 26B-A4B (Apache-2.0) two ways: through vLLM on an NVIDIA GPU, or in-process through MLX on Apple silicon. Both backends also serve text generation on an OpenAI-compatible /v1/chat/completions, at no extra GPU memory.

This fork adds three enterprise use cases, worked through with runnable code, and a write-up that makes the case for System One models in automation: https://stefletcher.github.io/openjev/. The rest of this README takes you from a clone to those examples running, then covers the API and the backends in depth. OpenJev is an independent project, not affiliated with or endorsed by TypeSafe AI.

Get started

1. Clone

git clone https://github.com/SteFletcher/openjev && cd openjev
python3 -m venv .venv && source .venv/bin/activate     # Python 3.10 or later

2. Start a server

Pick whichever matches your hardware. Either way the first start downloads the weights (about 16–18 GB) into ~/.cache/huggingface, and after that nothing leaves the machine.

Apple silicon (any M-series Mac with 32 GB or more; no Docker, no vLLM):

pip install -e '.[mlx]'
OPENJEV_BACKEND=mlx python -m openjev     # http://127.0.0.1:8080

NVIDIA GPU (24 GB or more, Docker with the NVIDIA container toolkit):

docker compose up -d                      # http://127.0.0.1:8080 once the model has loaded

No hardware? Codiv hosts OpenJev for free (100M System One tokens, no card required) at https://api.codiv.ai/v1/systemone. Use that as the base URL below.

If port 8080 is taken, set OPENJEV_PORT. The backends section has the minimum spec, throughput figures and every setting.

3. Make a decision

curl http://127.0.0.1:8080/v1/models     # lists openjev-latest once the server is up

curl http://127.0.0.1:8080/v1/systemone -H "Content-Type: application/json" \
  -d '{"model": "openjev-latest", "state": "I was charged twice this month.",
       "questions": {"is_billing": {"type": "noul", "instructions": "Is this a billing issue?"}}}'

Or with TypeSafe's SDK, which works against OpenJev unchanged:

pip install typesafe-sdk
export TYPESAFE_BASE_URL=http://127.0.0.1:8080   # or https://api.codiv.ai
export TYPESAFE_API_KEY=local                    # any value for a local server without auth

from typesafe_sdk import TypeSafeClient

client = TypeSafeClient()
r = client.system_one(
    "Everything is down and we have a demo with our biggest client at noon.",
    {
        "urgent": {"type": "noul", "instructions": "Does the customer need a reply within the hour?"},
        "team":   {"type": "choice", "instructions": "Which team should handle it?",
                   "criteria": {"outage": "service down", "billing": "charges, refunds", "feature": "requests, how-to"}},
        "tone":   {"type": "score", "instructions": "How upset is the customer?",
                   "criteria": ["calm", "annoyed", "furious"]},
    },
)
r.nouls["urgent"].noul        # 1.00
r.choices["team"].choice      # "outage", confidence 1.00
r.scores["tone"].score        # 2.00 (expected level, 0-indexed)

4. Run the examples from the write-up

The write-up works through three enterprise problems: evals with a judge that sends unsure cases to a person, personal data in logs that regexes miss, and routing each log event to the cheapest model that can handle it. Each is a graph where System One reads sit on the edges and decide where an item goes next. The code comes at two levels.

The demos (examples/) use only the standard library. Try them offline first, then against the server you just started:

python examples/evals.py --mock                          # answers come from keyword hints
python examples/pii_in_logs.py --mock
python examples/model_routing.py --mock

export OPENJEV_BASE_URL=http://127.0.0.1:8080            # now the real model
python examples/evals.py
python examples/pii_in_logs.py
python examples/model_routing.py

The worked implementations (examples/worked/) are how each would be built for real: TypeSafe's SDK for the reads, LangGraph for the graphs and the Anthropic SDK for the Claude steps.

pip install -e '.[examples]'                    # typesafe-sdk, langgraph, anthropic
export TYPESAFE_BASE_URL=http://127.0.0.1:8080  # set in step 3
export TYPESAFE_API_KEY=local
export ANTHROPIC_API_KEY=...                    # only the guard, the router and the answer generator call Claude

# evals: grade a labelled set, with a CI-style gate
python examples/worked/evals/run_eval.py examples/worked/evals/dataset.jsonl --min-agreement 0.85

# PII: a streaming filter, safe lines out on stdout, the unsure ones to a quarantine file
python examples/worked/pii/log_filter.py --quarantine quarantine.log < examples/worked/pii/sample.log

# routing: each event to Haiku, Sonnet, Opus or Fable, checked and escalated (spends a few cents)
python examples/worked/routing/run_router.py examples/worked/routing/events.log

The first two need only the System One server. Each folder's README and the write-up's How part go through the code step by step, and the code blocks on the site are copied from these files, so what you read is what runs.

All three run offline in the tests against a fake server and a fake Claude (pip install -e '.[test]' && pytest). That proves the plumbing, not the answers: build a labelled set for each decision before you trust a threshold.

API

Question types:

• noul (yes/no): takes optional criteria: {true, false} and returns {noul: P(yes)}.
• choice: takes criteria: {name: description} and returns {choice, probabilities, confidence}.
• score: takes criteria: [level0, level1, …] (2–10 levels) and returns {score: Σ i·pᵢ, legend, probabilities, confidence}.

confidence is 1 − H(p)/ln K. It is 1 when the model is certain and 0 when the distribution is uniform. usage.input_tokens counts prompt tokens, including image tokens. usage.output_tokens is 0 unless you set think (below).

Errors follow the same shapes as Jev, checked against the live API:

• 422 with a FastAPI validation list for a field of the wrong shape.
• 400 with the reason as plain text for a question the server cannot ask (no options, too many options, or too many score levels).
• 400 api_usage_error for an unknown model or question type.
• {"detail": {"error_type", "message"}} for auth errors (401/403).
• 429 for rate limits, and 529 when the server is overloaded.

Known differences from Jev:

• A choice can have at most 128 options. Jev allows 255. The refusal reads the same.
• Model names are OpenJev's own. jev-latest and jev-preview are aliases. A pinned Jev version such as jev-1.13.0 answers 400 Unknown model.
• A choice with one option, or a score with one level, gets a direct answer (probability 1) with no read, so it bills no tokens. Jev returns the same values and bills for the read.
• A body nested a thousand levels deep answers 422. Jev answers 500.
• The server answers many questions in chunks of about 12 per read, still in parallel.

Extensions

These optional request fields are OpenJev additions to Jev's contract. Leave them out and a request behaves exactly like Jev. TypeSafe's SDKs never send them. They come from the example server in vllm-project/vllm#57250.

curl https://api.codiv.ai/v1/systemone \
  -H "Authorization: Bearer $TYPESAFE_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "openjev-latest", "state": "Look at the photo.",
       "images": ["data:image/jpeg;base64,/9j/4AAQ..."],
       "questions": {"hotdog": {"type": "noul", "instructions": "The photo shows a hot dog"}}}'

think and sequential need a text state, so you cannot combine them with images. Such a request gets a 400.

Text generation

POST /v1/chat/completions generates text with the same DiffusionGemma, OpenAI style. Tools that talk to a chat model can point their base URL at OpenJev. Use model diffusiongemma-26b. Both backends serve it, streaming and not, and a client cannot tell them apart except where this section says so.

vLLM refuses some fields for diffusion models, so OpenJev adjusts requests instead of failing them:

• It ignores temperature, seed, min_p, logit_bias, penalties and reasoning.
• response_format (json_object or json_schema) becomes an instruction to reply with JSON only. The server returns the first JSON object in the reply as the content.
• max_tokens defaults to 1024, with a cap of 8192. Thinking stays off unless you set chat_template_kwargs: {"enable_thinking": true}. The thought then comes back separately in message.reasoning, never in content.
• stream: true streams server-sent events and always ends with a usage chunk.
• The endpoint supports tools (tools, tool_choice).

On MLX, three of these differ. tools, tool_choice, logprobs and top_logprobs are accepted and ignored, not answered. A stop string that is more than one token is dropped, because only a single token can end a denoised block. And there is no message.reasoning: the model's thought channel is stripped from the reply, so enable_thinking changes the answer but does not return the thought.

Generation denoises the output in 64-token blocks, so it costs far more GPU time than a System One read. At most 8 generations run at once, so they never crowd out reads. On MLX the model runs on one thread, so generations queue rather than overlap, and a client that disconnects stops the reply at the next block instead of the next token.

from openai import OpenAI

client = OpenAI(base_url="https://api.codiv.ai/v1", api_key="sk-codiv-...")
r = client.chat.completions.create(
    model="diffusiongemma-26b",
    messages=[{"role": "user", "content": "Summarize: 3 nonstop flights, cheapest $212 on Delta."}],
)
print(r.choices[0].message.content)

How it works

DiffusionGemma is a discrete diffusion model. It denoises a whole canvas of tokens per forward pass instead of generating left to right. OpenJev uses that to read answers instead of writing them.

It builds a canvas where the only masked tokens are the answer slots, one token per question:

canvas in                 one read-only pass         answer out
  q1: [?]        ──►      P(yes) 0.001        ──►    noul  0.001
  q2: [?]                 P(A) 0.000                 choice "billing"
                          P(B) 0.999                 confidence 0.997
                          P(C) 0.000
  q3: [?]                 P(0) 0.000                 score 1.00
                          P(1) 0.996
                          P(2) 0.004

Every label is a single token: yes/no for a noul, A/B/C for a choice, 0/1/2 for a score. The model never writes into those slots. One read-only pass returns the probability distribution over each slot, and that distribution is the answer. The numbers above are a real read of "The invoice looks wrong again. Second time this quarter.": not urgent, billing, mildly annoyed.

Two consequences follow. An answer cannot go off-schema, because the read scores only the label tokens. And the confidence comes from the model's own distribution, not from a number the model reports about itself.

If any slot is uncertain (entropy > 0.1), OpenJev re-reads with fresh noise up to four times and averages the results. Question ids never reach the model: it sees q1, q2, q3.

The vLLM side of this is vllm-project/vllm#57250, which adds seeded canvases, read-only steps and step caps for DiffusionGemma. openjev/engine.py is adapted from that PR's structured_server.py example. It adds async I/O, bounded concurrency and backpressure.

Backends in depth

Two backends serve the same /v1/systemone. Pick by hardware:

NVIDIA GPU in depth

You need an NVIDIA GPU with at least 24 GB of memory for the NVFP4 checkpoint (tested on an RTX PRO 6000 Blackwell, sm_120). The weights are about 18 GB.

A prebuilt image is on Docker Hub, so there is nothing to compile. razorback16/openjev runs vLLM with PR #57250 in one container with the Jev-compatible API server. It uses CUDA 13 and the pin below.

docker compose up -d          # OpenJev on 127.0.0.1:8080 once the model has loaded
curl localhost:8080/v1/models

Or without compose:

docker run -d --gpus all --ipc=host -p 127.0.0.1:8080:8080 \
  -v ~/.cache/huggingface:/root/.cache/huggingface razorback16/openjev:0.3.0

The model weights (about 18 GB) download on first start into ~/.cache/huggingface. Use docker compose build to build the image yourself instead. vLLM listens only inside the container. Set OPENJEV_UPSTREAM to skip it and use a vLLM server you already run.

Measured on an RTX PRO 6000 using 38% of the GPU, with 3 questions per request and cache-busted states:

Without Docker:

git clone https://github.com/razorback16/vllm && cd vllm
git checkout baa833874881ba62cef99e0c5b716fb136c4a009   # the same commit the image pins
VLLM_USE_PRECOMPILED=1 \
  VLLM_PRECOMPILED_WHEEL_COMMIT=36fa72d2d0d2f86c7c83e1e99c9012b7bd26463b pip install -e .
vllm serve nvidia/diffusiongemma-26B-A4B-it-NVFP4 --served-model-name dgemma \
  --diffusion-config '{"canvas_length": 64}' --max-logprobs 32 --enable-prefix-caching \
  --async-scheduling --attention-backend TRITON_ATTN \
  --limit-mm-per-prompt '{"image": 8, "video": 0}' \
  --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4 \
  --override-generation-config '{"max_new_tokens": null}'
pip install -e path/to/openjev && python -m openjev

Apple silicon in depth

OPENJEV_BACKEND=mlx runs DiffusionGemma inside the OpenJev process through MLX and mlx-vlm. The 4-bit weights are 16.6 GB and need about 16 GB of GPU-addressable memory, which on macOS means a 32 GB machine in practice; macOS 14 or later. The 8bit and bf16 builds exist too (OPENJEV_MLX_MODEL).

/v1/systemone answers reads with the same prompts, canvases and seeds as the vLLM backend, including images, samples, sequential, steps and the automatic re-reads. Each denoise step past the first reuses the one prefill of the prompt and writes back only the answer slots, so the template cannot drift: more steps cost GPU time, not prompt tokens. think writes the thought with mlx-vlm's own denoise loop, then reads after it, and bills exactly as vLLM does.

An image read builds its prompt with the mlx-vlm processor, which expands each image into its soft tokens. usage.input_tokens and OPENJEV_MLX_MAX_PROMPT both count that expanded prompt. The server keeps the prefill of a recent prompt, so the re-reads and the samples of one request share a single vision pass.

Reads run one at a time, so this backend suits local use rather than serving. A 3-question request takes about 0.2–0.4 s on an M3 Ultra and about 0.39 s on an M4 Max, both with the 4-bit weights. 16 concurrent requests finish at about 4 req/s.

OPENJEV_MLX_TEST_MODEL=path/to/weights pytest tests/test_mlx_model.py   # tests against the real model

Settings

The server reads its settings from the environment.

Caveats

• vllm-project/vllm#57250 has not merged yet. The request fields it uses (vllm_xargs) are provisional, so this project pins a commit (razorback16/vllm branch structured-reads-57250-rebased). That branch is the PR's head plus three fixes that keep vLLM's engine from crashing:

  • vllm-project/vllm#54309, for any request with an image (vllm-project/vllm#56712).
  • The sampler step fell back to eager code with a dtype mismatch once torch.compile hit its recompile limit. This broke multi-step reads and text generation.
  • Logprobs stashed at different steps could not join when reads and generations shared a batch.

  The branch also carries vllm-project/vllm#57416, which merged upstream after the PR's base. Without it, a prefill-only batch gets the wrong number of logit rows.

• Answer quality is the quality of DiffusionGemma 26B-A4B used in this mode. Evaluate it on your own tasks before you rely on it.

Development

pip install -e '.[test]' && pytest              # includes the examples, offline
python site/snippets.py                         # re-copy the code blocks and contents into site/index.html

site/ is published to GitHub Pages on every push to main. The build fails if the page's code blocks have drifted from examples/worked/.

License

Apache-2.0. The DiffusionGemma weights are Apache-2.0 (NVIDIA / Google).