FastJev is an open source implementation of Jev for deployment on infrastructure you control. It continues SemIf as an independently maintained fork. Pinned open models run the Choice, Boolean, and Score interface through Torch, vLLM, MLX, llama.cpp, and optional EXL3 quantization. The WebGPU demo provides browser local inference.

Why FastJev?

FastJev turns Jev deployment into a standard Python workflow. The SDK loads the model, validates 2 to 16 options, performs scoring, and returns typed results with probabilities, token usage, timing, model revision, and prompt version.

Resident Torch, batched vLLM, native MLX, llama.cpp GGUF, EXL3, the CLI, and the optional System One compatible HTTP API share the same result model. Torch, MLX, llama.cpp, and EXL3 return results in one scoring pass with zero output tokens.

Each result records the model revision and prompt version. Published evaluations add row data, raw timings, and checksums for reproducibility.

Models you can run now

The same native BF16 direct logit interface has been validated on these pinned checkpoints. Put any listed model ID and revision into the quick start below.

The table reports native BF16 checkpoint quality. Browser artifacts use separate quantized formats. Their smoke results, exact rows, and revisions are in the model ladder report.

Quick start

The default backend requires Python 3.10+ and either exactly one visible CUDA GPU or Apple Silicon MPS. device="auto" prefers CUDA and otherwise selects MPS. Qwen3.5-4B uses about 9 GB of disk for the source checkpoint and measured 7.891 GiB peak allocated GPU memory in the historical RTX 5090 SDK smoke run.

python -m venv .venv
. .venv/bin/activate
export HF_HOME=/path/to/large-drive/huggingface
pip install 'fastjev[torch]'

The first call automatically downloads the pinned model revision from Hugging Face and caches it under HF_HOME.

from fastjev import Choice, FastJev, Option

with FastJev.from_pretrained(
    "Qwen/Qwen3.5-4B",
    revision="851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
) as jev:
    result = jev.decide(
        state={"message": "I was charged twice and need a refund today."},
        question=Choice("Which queue should handle this request?", [
            Option("access", "Account access and authentication."),
            Option("billing", "Billing, payments, and refunds."),
            Option("sales", "Pricing and new contracts."),
        ]),
    )

print(result.value)
print(result.probabilities)
print(result.provenance)

Remote models use an immutable Hugging Face revision containing 40 characters. Local model directories use a descriptive revision label for result provenance.

Install fastjev[llama-cpp] to run GGUF files from disk or Hugging Face. The Python SDK guide covers Apple Silicon, EXL3, llama.cpp prefix reuse, calibration, and provenance.

Batch multiple states

Use decide_batch to apply the same questions to multiple inputs:

from fastjev import Boolean, FastJev

with FastJev.from_pretrained(
    "Qwen/Qwen3.5-4B",
    revision="851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
    batch_size=8,
    sort_by_length=True,
) as jev:
    results = jev.decide_batch(
        ["Please refund the duplicate charge.", "I cannot log in."],
        {"refund": Boolean("Does the customer request a refund?")},
    )
    print([item["refund"].value for item in results])

Torch batching is opt-in; the default batch_size=1 retains sequential scoring. The size counts decision prompts, not states. Results retain input order, but batch shape can change reduced-precision probabilities and close decisions. This is a Python SDK feature; the CLI and HTTP service do not automatically batch multiple states. See the SDK guide and measurements and limitations.

Install the latest source

Use an editable checkout to run the latest code from main.

git clone https://github.com/chengyongru/fastjev.git
cd fastjev
pip install -e '.[torch]'

Measured results

RTX 5090 backend comparison

All three measurements used Qwen3.5-4B through the public decide_many API on one RTX 5090 under WSL2, with one warmup and seven measured three-question calls.

On this RTX 5090 / WSL workload, vLLM delivered 1.96× Torch throughput and 48.9% lower median batch latency. Its additional 30.38 seconds of startup cost breaks even after roughly 383 batches of three decisions when the process stays resident. Both backends made the same three selections.

PR #7 records this historical integration measurement, including the exact environment and aggregate medians. The repository's reproducible benchmark bundle covers separate experiments with committed row data.

The llama.cpp row uses a later shell-safeguard fixture with 183, 185, and 165 input tokens, while the Torch/vLLM rows share identical 125, 152, and 151-token prompts. It shows practical SDK cost, but the different prompts preclude an exact llama.cpp-to-BF16 speed ratio. All llama.cpp decisions selected the expected answers with zero output tokens. See the results report for its quality results, exact conditions, and row-level evidence.

The optional 27B EXL3 backend was validated separately on the same RTX 5090 with the same three-question safeguard: all selections matched, output token count remained zero, and the median call took 365.48 ms. It is excluded from the same-model table because model size, quantization, runtime, and background GPU allocation differ. The results report also records the calibrated Torch smoke and llama.cpp prefix-reuse comparison.

Decision quality

The first two rows report balanced accuracy; the authored metric averages the three decision families equally. The third reports modal agreement with equal weighting across cases. The EXL3 value uses the same authored rows and metric, but it is a system-level comparison: model family, size, quantization, and runtime all differ. The Jev column reproduces public records. Open-model values come from local frozen evaluations. The results report provides the full data, perturbations, and claim boundaries.

How it compares

These projects serve adjacent deployment needs. Their published performance belongs to their own workloads.

Backends and interfaces

Use fastjev-score for JSONL jobs. Install fastjev[api,torch] and run fastjev-serve for POST /v1/systemone and GET /v1/models. The SDK guide covers batching and custom backends. The HTTP guide covers server setup, authentication, and compatibility boundaries.

Integrations should import fastjev and use the fastjev-* commands. Only the documented modules are part of the supported public API.

Documentation

The results report covers speed, quality, perturbations, and limitations. The calibration report covers workload-scoped temperature scaling. The method guide documents frozen prompts, metrics, and timing scope. The reproduction guide provides pinned environments and verification commands. The benchmark bundle contains fixtures, runners, and source selection. The interactive replay, verified Jev Ultrafast browser demo, and WebGPU browser demo provide visual ways to explore the project.

Boundaries and provenance

FastJev returns probabilities conditioned on the supplied options. They remain calibrated=False unless an identity-bound TemperatureCalibration is explicitly attached. Calibration fitted for another workload, backend, model revision, or prompt version must not be reused. The experimental Torch shared-prefix modes can change close BF16 argmax results; llama.cpp prefix reuse is also explicit and disabled by default.

Model weights stay on upstream hosts, and third party evaluation records stay with their original sources. Upstream models retain their licenses. Exact revisions are listed in THIRD_PARTY.md.

The third party record documents source and model provenance.

Project code is released under the MIT License.