Decision models answer typed questions about a state (text, JSON, images, video) with a probability for every allowed option, in one forward pass and without generating text. TypeSafe's Jev introduced the /v1/systemone API; open-weight models such as Clef-Flash now follow it. These models carry a custom scoring head, so vllm serve alone cannot run them. This project serves them behind the same API, fast and ready for production.

Highlights

• Jev-compatible API. choice, score and noul questions, Jev's limits, images and videos. Clients written for Jev work unchanged.
• vLLM backbone + the model's own head. The backbone runs as a vLLM pooling model; the decision head runs in the same process on vLLM's hidden states. Concurrent requests are micro-batched.
• Verified against the reference. vLLM's token sequence is checked token by token against the model's own encoding, and answers match the reference transformers implementation (max total-variation distance 0.0067).
• Bugs fixed that upstream code has. Videos no longer collapse to 2 frames; video requests on the vLLM path work.
• Production details. Start-up warm-up, one error format, optional API key, systemd unit, Dockerfile, install script for CUDA 12 drivers.
• Tested. Unit tests in CI, an 87-check end-to-end suite, and reproducible benchmarks.

Supported models

More open Jev-like models are planned. Request one or add one.

Performance

Clef-Flash on one A800-SXM4-80GB (full results and setup):

The reference column already includes the linear-attention fast path; without it short requests take ~160 ms.

Quick start

Requirements: Linux, Python 3.10+, an NVIDIA GPU with a CUDA 12 or newer driver. Validated on an 80 GB A800; the bf16 weights take ~19 GB.

git clone https://github.com/Arcobalneo/open-jevlike-infer.git && cd open-jevlike-infer

# 1. Install. NVIDIA driver >= 580 (CUDA 13):
pip install -e ".[vllm]"
#    Driver 525-579 (CUDA 12): installs the CUDA 12.9 builds of vLLM and torch into .venv
pip install uv && scripts/install_cuda12.sh .venv && source .venv/bin/activate

# 2. Download the weights (~19 GB, sha256-verified)
python scripts/download_model.py --model clef-flash --dir ./models/clef-flash

# 3. Serve (ready after ~100 s of loading and warm-up)
jevlike-infer serve --model clef-flash --model-path ./models/clef-flash --port 8000

Ask a question:

curl -s http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d '{
  "model": "clef-flash",
  "state": "Our checkout started returning errors and orders are blocked.",
  "questions": {
    "department": {"type": "choice", "instructions": "Which team should handle it?",
                   "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}},
    "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},
    "outage":  {"type": "noul", "instructions": "Is a service down?"}
  }
}'

{
  "model": "clef-flash",
  "answers": {
    "department": {"type": "choice", "choice": "technical", "confidence": 0.9555,
                   "probabilities": {"billing": 0.0445, "technical": 0.9555}},
    "urgency": {"type": "score", "score": 1.8209, "confidence": 0.8793,
                "legend": {"0": "Can wait", "1": "This week", "2": "Today"},
                "probabilities": {"0": 0.0585, "1": 0.0622, "2": 0.8793}},
    "outage": {"type": "noul", "noul": 0.8105}
  },
  "usage": {"input_tokens": 299, "output_tokens": 0}
}

Images and videos go in images / videos as data URIs, base64, URLs, or (for video) lists of frames. See the API reference and examples/.

Deployment

• systemd: deploy/systemd/jevlike-infer.service
• Docker: deploy/docker/Dockerfile (CUDA 12.9 build, driver >= 525) and compose.yaml

Every option is a flag or a JEVLIKE_* environment variable (jevlike-infer serve --help):

How it works

request ─► validate (Jev limits) ─► decode media ─► micro-batch ─► vLLM backbone (pooling)
        ◄─ answers + usage ◄────── joint head on hidden states ◄────────────┘

The model's own encoder fixes where every question and option sits in the token sequence. Media placeholders are collapsed so vLLM can expand them again, vLLM runs the backbone and returns every token's hidden state, the result is checked token by token against the model's encoding, and the model's own head scores the options. Details: docs/architecture.md.

Compared with existing options

Testing

pip install -e ".[dev]" && pytest                               # unit tests, no GPU
python tests/e2e/run_e2e.py --base-url http://127.0.0.1:8000    # 87 checks against a running server
# server in Docker: add --fixture-host 172.17.0.1 so it can fetch the suite's test media

Benchmarks: benchmarks/.

Documentation

• API reference
• Architecture
• Clef-Flash notes
• Troubleshooting
• Adding a model

Contributing

Issues and pull requests are welcome, especially new models and results on other GPUs. See CONTRIBUTING.md.

Acknowledgements

• Cloudflare for releasing Clef and Clef-Flash under Apache-2.0.
• simonlehmann/clef-NVFP4 for the vLLM pooling approach this project builds on.
• vLLM.
• TypeSafe AI for the Jev / SystemOne API design.

This project is independent and not affiliated with Cloudflare or TypeSafe AI.

License

Apache-2.0. Model weights are downloaded from their own repositories under their own licenses; see NOTICE.