Fast structured decisions on Linux and Apple Silicon

Documentation · Demos · Getting Started · Supported Models · Example · Updates

Demos

🎉 Celebrating RadixFlow! (experimental)

https://github.com/user-attachments/assets/d99f5a8d-d715-4a79-8870-e56a6e48c073

Our first PR, which explored shared-prefix reuse and cache-on/off behavior, inspired us to look more closely at what happens after a cache hit. Reuse matters especially for prefill: Jev scores candidate branches through prefill, so reusing their common prefix avoids repeated computation.

We found a scheduling gap: a matching prefix can still have hybrid state being produced in the current round. Stopping the scan at that pending branch leaves room that independent ready branches could use. RadixFlow defers the pending branch, continues selecting ready work, and rechecks it after completion. The engineering integration recovered executable capacity in the tested hybrid models; the pressure comparisons report the resulting worker rounds, speed and token reuse.

RadixFlow is experimental and disabled by default. Enable it with --radixflow true on supported configurations.

We welcome bug reports, feedback, and discussions—please share your experience with RadixFlow!

Video understanding: live comparison

The same Valen checkpoint answers 24 structured questions about one video, using eight sampled frames and one A800 per runtime. In this live run, Valen's PyTorch reference path took 5.31 s, while vLLM Jev took 1.22 s (4.3× faster); all 24 selected answers agreed. Model loading is excluded. Watch the MP4.

Apple Silicon: live image decisions

Valen 2B answers six GameQA Maze image questions locally on an Apple M5 using MLX. The display follows live HTTP responses. Watch the MP4.

About

vLLM Jev serves compatible Jev-style checkpoints through vLLM. Give it a question and candidate answers; it returns a label and a probability for each candidate.

• Native vLLM serving on Linux: scheduling, batching, compilation, KV cache, and metrics.
• Apple Silicon preview: run supported text models through MLX or PyTorch MPS, and Valen multimodal decisions through MLX.
• Structured decisions: Choice, Noul (yes/no), and Score (ordered levels) over HTTP.
• Multimodal inference: Valen, vjev, RSI-Jev v4.0-VL, JevAny Pointer, and Clef accept text and images. Valen and Clef also accept short videos; platform availability and request formats are model-specific.
• RadixFlow (experimental, opt-in): bounded admission and state-ready traversal fill shared-prefix execution rounds. See configuration and pressure-test results.
• Automatic setup: start a supported Hugging Face model ID with one command.

Getting Started

On Linux with an NVIDIA GPU, install vLLM Jev with uv from the repository directory:

uv pip install .

For development, use an editable installation.

Start a server with a supported Hugging Face model ID:

vllm-jev serve ZefanCai/Open-Jev-2B

The command downloads and prepares the model, then starts native vLLM at http://127.0.0.1:8795. Later runs reuse the prepared checkpoint.

On an Apple Silicon Mac with macOS 15 or newer, set up once from the repository directory, then serve:

source scripts/install_mac.sh
vllm-jev serve Valen-Team/Valen-Preview-0923

The command downloads and prepares the model on first use. Send text or images to /v1/systemone for Choice, Noul, and Score decisions. For another Mac model, replace the model ID with a supported one from the table below.

CUDA_VISIBLE_DEVICES=0 vllm-jev serve ZefanCai/Open-Jev-2B \
  --gpu-memory-utilization 0.90 --port 8795

Visit our documentation to learn more.

• Installation
• Quickstart
• List of Supported Models

Supported Models

Choose a checkpoint for your platform and run its command. Linux uses native vLLM; Apple Silicon uses MLX, or PyTorch MPS for Laya.

The RSI-Jev Linux adapter was contributed by Shanghua Gao (PR #2), and the Clef adapter by Arcobalneo (PR #4). See the RSI-Jev and Clef guides for input limits and request formats.

The JevAny Pointer adapter was contributed by heshien / AsirAsir02 (PR #9). See the JevAny guide for requests and limits.

Additional decision models (experimental)

These text-only checkpoints use the same command and /v1/systemone API for Choice, Noul, and Score. Listed platforms have been tested.

JevK5 currently supports 2–16 options per question. In concurrent Linux batches, Kev-0.8B can change its selected label; This-That and Decider-2B can also change the most likely Score level. These consistency limits remain under investigation.

The optional CUDA setting VLLM_JEV_BF16_MATMUL=no_splitk can reduce batch-dependent numerical drift. Kev label changes can remain, including on A800. See BF16 precision for usage and performance tradeoffs.

See the model guide for limits and request examples.

Example

With the server running, send a Choice request:

curl -sS http://127.0.0.1:8795/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{"state":"I was charged twice and want a refund.","questions":{"intent":{"type":"choice","instructions":"Choose the customer intent.","criteria":{"billing":"A payment or refund issue","technical":"A malfunction or setup issue","other":"Another request"}}}}'

The response contains answers.intent.choice and answers.intent.probabilities for the supplied labels.

See the user guide for supported models, serving options, and Choice, Noul, and Score examples. The plugin targets vLLM 0.29.0 and Python 3.12+.

For image questions, start a supported vision model and follow its image request example. Short MP4 videos are supported by Valen on Linux and Mac, and by Clef on Linux.

RadixFlow pressure comparison

Updates

2026-10-14

• Added the inline RadixFlow explainer video and the story behind its cache-to-scheduling investigation. RadixFlow is explicitly experimental and disabled by default.
• Added --radixflow true|false with --radixflow / --no-radixflow aliases, shared by Jev and ordinary generative-model serving. Existing policy flags remain compatible.
• Added reusable Rust-library configuration through VLLM_JEV_RADIXFLOW_LIBRARY. The integration is model-independent within supported native prefix-cache configurations; incompatible cache/scheduler settings are rejected explicitly. Usage.

2026-10-13

• Added opt-in RadixFlow scheduling for synchronous Linux serving with prefix caching. The default serving path is unchanged; setup and supported configurations.
• Added three-run pressure comparisons on Open-Jev-2B and Open-Jev-9B, plus an OpenJev-0.6B compatibility control. Results include worker-round counts, actual token reuse and selected-answer agreement. Results and measurement details.
• Added Linux text/image serving for JevAny Pointer 4B and 27B, contributed by heshien / AsirAsir02 (PR #9).
• Fixed automatic model preparation and explicit protocol selection. JevAny supports all six context_order permutations while preserving its Pointer readouts; native remains the default.
• Verified serving, image handling, prompt ordering and cancellation recovery with regression checks and real-model stress tests.

Inference performance

Contributing

Report bugs and feature requests in Issues. Use Discussions for questions and ideas. Submit code changes as pull requests; the maintainer reviews them before merging.

License

Apache-2.0. The Choice prompt follows MIT-licensed Open-Jev; see third-party notices.

Citation

If you use vLLM Jev in your research, please cite our paper: RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving.

@misc{jing2026radixflow,
  title         = {{RadixFlow}: Filling Execution Bubbles in Prefix-Cached Hybrid {LLM} Serving},
  author        = {Huihao Jing and Wenbin Hu and Shaojin Chen and Haochen Shi and Haoyu Huang and Sirui Zhang and Yuxuan Liu and Zhongwei Xie and Haoran Li},
  year          = {2026},
  eprint        = {2610.18143},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2610.18143}
}