A mixed NVFP4 / FP8 quantization of Cloudflare/clef, the 27B multimodal decision model. Unofficial — not affiliated with or endorsed by Cloudflare.
Clef answers typed questions about a state (text, JSON, images, video) with one probability per allowed option, in a single prefill pass. Its value is the probabilities, so this quant was checked option by option against the BF16 release, not only on accuracy.
23 GB on disk (BF16 release: 55 GB); 20.2 GiB of weights in vLLM.
| Part | Format |
|---|---|
MLP gate/up/down_proj, layers 0–55 | NVFP4 (W4A4, group 16, FP8 scales, static activation global scale) |
MLP gate/up/down_proj, layers 56–63 | FP8 (per-channel weights, dynamic per-token activations) |
Full-attention q/k/v/o_proj; linear-attention in_proj_qkv/in_proj_z/out_proj | FP8 (as above) |
Linear-attention in_proj_a/in_proj_b, norms, conv, embeddings | BF16 |
| Vision encoder | BF16 |
lm_head | BF16 — the joint head reads its rows directly as option embeddings |
Joint schema head (joint_head.safetensors) | BF16, unchanged from the release |
The layout follows the community recipe for Qwen3.8-27B with one change: lm_head stays BF16.
The last eight layers stay FP8 on purpose: comparing the release against
Qwen/Qwen3.8-27B, the vision encoder and early layers
are byte-identical, and Clef's post-training lives in the upper layers.
Method: llm-compressor QuantizationModifier (round-to-nearest, no GPTQ), calibrated on 374
Clef-format records — 128 BANKING77 intent questions (77-way and 10-way schemas), 150 game-agent
state records with a choice question, and 96 Flickr30k photos with noul/choice/score
questions. Calibration and evaluation records are disjoint. recipe.yaml is the exact recipe.
Same 615 records through the BF16 release and through this checkpoint on vLLM's real NVFP4/FP8
kernels (FlashInfer CUTLASS FP4 GEMM). agreement = top option identical to BF16. TV = mean
total-variation distance between the per-question distributions (0 = identical, 1 = disjoint).
| Eval set | Questions | Agreement | Mean TV | Accuracy BF16 → NVFP4 |
|---|---|---|---|---|
| BANKING77 test, full 77-way schema | 400 | 98.5% | 0.019 | 94.25% → 94.50% |
| Flickr30k photos, 4 mixed-type questions each | 256 | 96.9% | 0.017 | (no labels) |
Game-agent states, 2–4 option choice | 150 | 96.7% | 0.047 | 68.7% → 67.3% * |
| README invoice example | 2 | 100% | 0.0007 | — |
* graded against synthetic labels that the BF16 model itself matches only 69% of the time; read the agreement column, not the accuracy, for that row.
Disagreements are near-ties: in the calibration-time (fake-quant) run, the 15 flipped questions had a median BF16 top-1/top-2 margin of 0.13 (11 of 15 under 0.2), against 0.95 for questions that agreed.
One decision at a time (batch 1, the latency case) on an NVIDIA DGX Spark (GB10, 273 GB/s unified memory), vLLM 0.23.1, CUDA graphs on:
| Record | Input tokens | Median latency |
|---|---|---|
| text state, 1 question | 288 | 177 ms |
| 640 px image + 4 questions | 621 | 257 ms |
| BANKING77, 77 options | 1834 | 705 ms |
The joint head adds 3–6 ms of that. On this machine the floor (~170 ms) is set by reading the weights once per pass; GPUs with more memory bandwidth should be considerably quicker (not measured here).
The files, record format and systemone API are the same as the BF16 release; see the
Clef model card for the full input format.
clef_vllm.py runs the backbone in vLLM as a pooling model (final hidden state of every token)
and Clef's joint head on top, in the same process. Needs a Blackwell GPU for NVFP4 and a vLLM build
with Qwen3_5ForConditionalGeneration (tested: 0.23.1 nightly).
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("simonlehmann/clef-NVFP4")
sys.path.insert(0, path)
from clef_vllm import ClefVLLM
clef = ClefVLLM(path, max_model_len=16384, gpu_memory_utilization=0.4)
response = clef.systemone({
"model": "clef",
"state": "Our checkout started returning errors and orders are blocked.",
"questions": {
"department": {
"type": "choice",
"instructions": "Which team should handle the message?",
"criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"},
},
"urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},
"outage": {"type": "noul", "instructions": "Is a service down?"},
},
})
print(response["answers"])
# Many records at once, images included (PIL), batched through vLLM:
# clef.probabilities([record, ...]) -> [{question_id: {option_id: probability}}, ...]
max_images / max_videos in the constructor set vLLM's per-request media limits (default 1 image).
transformers' compressed-tensors integration cannot run the mixed NVFP4/FP8 layers compressed, so
load with run_compressed=False. This decompresses to BF16 at load time (~57 GB of GPU memory, no
speed-up): useful to check outputs, not to save memory.
import sys
from huggingface_hub import snapshot_download
from transformers import CompressedTensorsConfig
path = snapshot_download("simonlehmann/clef-NVFP4")
sys.path.insert(0, path)
from joint_schema_model import load_release_model, systemone
model, processor = load_release_model(
path, device="cuda", quantization_config=CompressedTensorsConfig(run_compressed=False)
)
Requires compressed-tensors in addition to torch and transformers.
Apache-2.0, following Cloudflare/clef and
Qwen/Qwen3.8-27B. joint_schema_model.py,
joint_head.safetensors, LICENSE and the tokenizer/processor files are redistributed unchanged
from the Cloudflare release; clef_vllm.py and the quantized weights are new.
A mixed NVFP4 / FP8 quantization of Cloudflare/clef, the 27B multimodal decision model. Unofficial — not affiliated with or endorsed by Cloudflare.
Clef answers typed questions about a state (text, JSON, images, video) with one probability per allowed option, in a single prefill pass. Its value is the probabilities, so this quant was checked option by option against the BF16 release, not only on accuracy.
23 GB on disk (BF16 release: 55 GB); 20.2 GiB of weights in vLLM.
| Part | Format |
|---|---|
MLP gate/up/down_proj, layers 0–55 | NVFP4 (W4A4, group 16, FP8 scales, static activation global scale) |
MLP gate/up/down_proj, layers 56–63 | FP8 (per-channel weights, dynamic per-token activations) |
Full-attention q/k/v/o_proj; linear-attention in_proj_qkv/in_proj_z/out_proj | FP8 (as above) |
Linear-attention in_proj_a/in_proj_b, norms, conv, embeddings | BF16 |
| Vision encoder | BF16 |
lm_head | BF16 — the joint head reads its rows directly as option embeddings |
Joint schema head (joint_head.safetensors) | BF16, unchanged from the release |
The layout follows the community recipe for Qwen3.8-27B with one change: lm_head stays BF16.
The last eight layers stay FP8 on purpose: comparing the release against
Qwen/Qwen3.8-27B, the vision encoder and early layers
are byte-identical, and Clef's post-training lives in the upper layers.
Method: llm-compressor QuantizationModifier (round-to-nearest, no GPTQ), calibrated on 374
Clef-format records — 128 BANKING77 intent questions (77-way and 10-way schemas), 150 game-agent
state records with a choice question, and 96 Flickr30k photos with noul/choice/score
questions. Calibration and evaluation records are disjoint. recipe.yaml is the exact recipe.
Same 615 records through the BF16 release and through this checkpoint on vLLM's real NVFP4/FP8
kernels (FlashInfer CUTLASS FP4 GEMM). agreement = top option identical to BF16. TV = mean
total-variation distance between the per-question distributions (0 = identical, 1 = disjoint).
| Eval set | Questions | Agreement | Mean TV | Accuracy BF16 → NVFP4 |
|---|---|---|---|---|
| BANKING77 test, full 77-way schema | 400 | 98.5% | 0.019 | 94.25% → 94.50% |
| Flickr30k photos, 4 mixed-type questions each | 256 | 96.9% | 0.017 | (no labels) |
Game-agent states, 2–4 option choice | 150 | 96.7% | 0.047 | 68.7% → 67.3% * |
| README invoice example | 2 | 100% | 0.0007 | — |
* graded against synthetic labels that the BF16 model itself matches only 69% of the time; read the agreement column, not the accuracy, for that row.
Disagreements are near-ties: in the calibration-time (fake-quant) run, the 15 flipped questions had a median BF16 top-1/top-2 margin of 0.13 (11 of 15 under 0.2), against 0.95 for questions that agreed.
One decision at a time (batch 1, the latency case) on an NVIDIA DGX Spark (GB10, 273 GB/s unified memory), vLLM 0.23.1, CUDA graphs on:
| Record | Input tokens | Median latency |
|---|---|---|
| text state, 1 question | 288 | 177 ms |
| 640 px image + 4 questions | 621 | 257 ms |
| BANKING77, 77 options | 1834 | 705 ms |
The joint head adds 3–6 ms of that. On this machine the floor (~170 ms) is set by reading the weights once per pass; GPUs with more memory bandwidth should be considerably quicker (not measured here).
The files, record format and systemone API are the same as the BF16 release; see the
Clef model card for the full input format.
clef_vllm.py runs the backbone in vLLM as a pooling model (final hidden state of every token)
and Clef's joint head on top, in the same process. Needs a Blackwell GPU for NVFP4 and a vLLM build
with Qwen3_5ForConditionalGeneration (tested: 0.23.1 nightly).
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("simonlehmann/clef-NVFP4")
sys.path.insert(0, path)
from clef_vllm import ClefVLLM
clef = ClefVLLM(path, max_model_len=16384, gpu_memory_utilization=0.4)
response = clef.systemone({
"model": "clef",
"state": "Our checkout started returning errors and orders are blocked.",
"questions": {
"department": {
"type": "choice",
"instructions": "Which team should handle the message?",
"criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"},
},
"urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},
"outage": {"type": "noul", "instructions": "Is a service down?"},
},
})
print(response["answers"])
# Many records at once, images included (PIL), batched through vLLM:
# clef.probabilities([record, ...]) -> [{question_id: {option_id: probability}}, ...]
max_images / max_videos in the constructor set vLLM's per-request media limits (default 1 image).
transformers' compressed-tensors integration cannot run the mixed NVFP4/FP8 layers compressed, so
load with run_compressed=False. This decompresses to BF16 at load time (~57 GB of GPU memory, no
speed-up): useful to check outputs, not to save memory.
import sys
from huggingface_hub import snapshot_download
from transformers import CompressedTensorsConfig
path = snapshot_download("simonlehmann/clef-NVFP4")
sys.path.insert(0, path)
from joint_schema_model import load_release_model, systemone
model, processor = load_release_model(
path, device="cuda", quantization_config=CompressedTensorsConfig(run_compressed=False)
)
Requires compressed-tensors in addition to torch and transformers.
Apache-2.0, following Cloudflare/clef and
Qwen/Qwen3.8-27B. joint_schema_model.py,
joint_head.safetensors, LICENSE and the tokenizer/processor files are redistributed unchanged
from the Cloudflare release; clef_vllm.py and the quantized weights are new.