vllm-radiance (MXFP4)

A vLLM inference server for the AMD Radeon AI PRO R9700 (gfx1201 / RDNA4), packaged as a container image. It bundles a working ROCm + PyTorch + Triton + AITER + vLLM stack with the RDNA4 patches and custom kernels needed to run vLLM on this card, plus RDNA4-tuned GEMM / attention / all-reduce paths and a speculative draft controller.

One command after the clone gets you a server. You do not build an image, and you do not edit anything for a different card count.

git clone https://codeberg.org/ggz14/radiance-vllm-mxfp4 && cd radiance-vllm-mxfp4
./docker-quickstart.sh   # checks the host, fetches ~40 GiB, starts the server, sends a test request

Or drive the two scripts it wraps, on podman or docker:

./setup-mxfp4.sh      # host check, image pull, checkpoints, kernels (~40 GiB, mostly download)
./serve-mxfp4.sh      # serve on http://localhost:8080/v1

Either way you end up at http://localhost:8080/v1. Quick start walks the whole path, with what each step costs and what to do when one of them stops.

Contents

• Status
• Quick start
• Requirements
• Setup
• Checkpoints
• Running the server
• ParoQuant (int4 W4A8 and int5 W5A8)
• Configuration
• Troubleshooting
• Performance
• How the MXFP4 path works
• What's inside the image
• Building the image from source
• Repository layout
• Further reading

Status

Early dev, experimental. Not production hardened. Use at your own risk.

Repo version 0.12.0; pinned image stilldeadcode/vllm-radiance:0.9.3.

This repository carries the MXFP4 work on top of vllm-radiance, the source for the stilldeadcode/vllm-radiance image on Docker Hub. The launcher pulls that published image and applies this repo's patches and kernels at container start, so running the MXFP4 stack never requires building an image.

Quick start

What you need

Docker: one command

./docker-quickstart.sh

It runs the same two scripts as the manual path, adds the checks that only bite Docker users, and does not hand the terminal back until the server has answered a real request:

Interrupt it whenever you like: re-running skips every step already done and downloads resume. --yes skips the download prompt, --no-drafter skips the 2 GiB drafter, --port 8081 moves the port, --foreground runs the server in your terminal instead of behind it, and --help lists the rest.

Once it is up, the same script is the front end for everything routine:

Two things are true of Docker and not of podman, and both surface as confusing failures if nobody says them first. Your user has to be able to reach the daemon socket — sudo usermod -aG docker $USER then newgrp docker, which is what the script tells you if it cannot. And Docker runs containers as root, so the checkpoints and the compile cache end up root-owned on the host and want sudo to delete. Rootless podman has neither property, which is why it is the preferred runtime here; RUNTIME=podman ./docker-quickstart.sh gets the same guided run on it.

Or drive it yourself

docker-quickstart.sh is a wrapper, not a dependency. These are the two scripts it calls, and both take RUNTIME=docker or RUNTIME=podman:

./setup-mxfp4.sh      # idempotent: re-run any time, it skips whatever is already done
./serve-mxfp4.sh      # runs in the foreground; DETACH=1 puts it in the background

The launcher picks the tensor-parallel size from the cards it finds. To pin it, use the wrappers (each is one line: serve-mxfp4.sh with TP set, every other knob and argument passed through):

./serve-tp1.sh        # one card  -- the single-GPU profile (GPUS=1 to pick which card)
./serve-tp2.sh        # two cards -- the default on a two-card host
./serve-tp3.sh        # three cards, via dummy-head padding (explicit only; see TP=3 below)

Check it works

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.8","messages":[{"role":"user","content":"Hello!"}]}'

An OpenAI-compatible endpoint, so any client that speaks that API works against it unchanged. The served model name is Qwen3.8 (Qwen3.6 and Qwen3.8-MXFP4 are aliases for the same one).

What you get

Qwen3.8-27B in native 4-bit MXFP4 with an FP8 speculative drafter. On two R9700:

Full numbers in Performance. On one R9700 (./serve-tp1.sh, 65,536 context): combined single-stream decode 137.7 t/s at 35.7 ms per update, 416 t/s aggregate at 8 concurrent (487 at 16 with MAXSEQS=16), prefill 2.5-2.8k t/s -- see One card (TP=1).

Useful next commands

./serve-mxfp4.sh --help     # every knob, short form
./gpu-detect.sh             # what was detected and what it will do with it
DRY_RUN=1 ./serve-mxfp4.sh  # print the container command without running it

Any argument serve-mxfp4.sh does not recognise is passed straight through to vllm serve.

Serving the plain FP8 checkpoint instead, through docker compose, is its own short path.

Requirements

Setup

What setup-mxfp4.sh does

First start is slow

The first serve-mxfp4.sh after setup spends several extra minutes compiling Triton and inductor kernels before the engine comes up. It looks idle; it is compiling. The result is cached in ~/.radiance-cache-w4a8-093-gdnm and later starts skip it.

Checkpoints

setup-mxfp4.sh downloads the defaults on its own. Every repo id is a ${VAR:-default} override, so pointing at a different one needs no edit.

To serve the uncensored variant, download it under $MODELS and point the launcher at it. There is no rewrite step and nothing else changes:

SNAP=$HOME/models/Qwen3.8-27B-Uncensored-MXFP4-awq ./serve-mxfp4.sh

That one is verified from its config.json, which declares all 8 mtp.* modules fp8_e4m3 per-channel in layer_quant_config, exactly what fp8_mtp.py produces. It has not been load-tested here; every number in this README is the AMD checkpoint.

Why AMD's checkpoint needs a rewrite

Its mtp.* layers are bf16, and its exclude list does name them, but as tensor names (mtp.fc.weight, all 15 .weight-suffixed) among 112 module names (model.visual.blocks.0.attn.qkv). Quark matches modules, so mtp.fc.weight never matches the module mtp.fc, the exclusion silently does nothing, vLLM applies the mxfp4 scheme to a bf16 head, and the load dies with:

Attempted to load weight (torch.Size([5120, 10240])) into parameter (torch.Size([5120, 5120]))

So the check for any other MXFP4 checkpoint is not "is mtp mentioned" but "is it mentioned the same way as everything else": module names in exclude, or an entry in layer_quant_config. fp8_mtp.py fixes the AMD one by requantizing the head to fp8 and declaring it properly.

Running the server

GPU and TP detection

serve-mxfp4.sh works out three things for itself at startup, and all three are overridable.

./gpu-detect.sh

AMD GPUs usable:  2 x Radeon AI PRO R9700 (0x7551, 32624 MiB each)
HIP indices:      0,1
skipped (<8192 MiB): 2:0x13c0:2048MiB
tensor parallel:  2   (supported: 8 4 2 1)
hardware sig:     2x7551-32624
KV cache pin:     18563072000 bytes (17.29 GiB/GPU, measured)

Which GPUs. gpu-detect.sh reads mem_info_vram_total from sysfs for every amdgpu render node, with no rocm-smi on the host and no container start just to count cards. Anything below MIN_GPU_MIB (8192) is excluded, which is what keeps an integrated GPU out of the count: the development box reports three AMD render nodes (two R9700 and a 2 GiB Granite Ridge iGPU), and a naive count picks --tensor-parallel-size 3.

Tensor-parallel size. The largest of 8 4 2 1 the usable cards can fill. 3, 6 and 12 are not picked automatically even though they divide num_attention_heads=24, because TP must also divide the GDN layers' linear_num_key_heads=16 and linear_num_value_heads=48. A three-card host therefore serves on two by default, leaves one idle, and says so.

KV cache size. See KV cache calibration.

One card (TP=1)

TP=1 (or ./serve-tp1.sh) turns on the single-GPU profile (SINGLE_GPU_PROFILE=auto), a set of choices that only make sense when one card holds the whole model. Every one of them is gated on TP=1 so a two-card serve is byte-identical to before:

Which card: GPUS=1 ./serve-tp1.sh names it (the fix for a single non-zero card landed 2026-09-16; earlier launchers left the engine with no visible device). Two things to know:

• The NVFP4 checkpoint barely fits one card. Its bf16 lm_head puts the load at ~20 GiB and vLLM's profiler cannot reach the 2.85 GiB the 65k cache needs; it serves only under a pin, which is what the shipped TP=1 row provides (6.09 GiB, ~140k KV tokens). The native MXFP4 checkpoint has no such trouble (~99k tokens at 8 sequences, ~85k at 16).
• More streams: MAXSEQS=16 ./serve-tp1.sh measured 487 t/s aggregate at 16 concurrent against 416 at 8 (TTFT p50 0.5 s). It re-profiles KV (the pin row is keyed on MAXSEQS=8); run MAXSEQS=16 ./calibrate-kv.sh to claim the margin there.

Numbers in One card (TP=1) under Performance; the engineering ledger is in PERFORMANCE.md (rows dated 2026-09-16/17).

TP=3 (explicit)

Three cards are served through zero-weight dummy heads. radiance_tp3pad.py widens the config to 36 q / 6 kv / 18 GDN-key / 54 GDN-value heads, MLP 17472 and vocab 248448, and pads the checkpoint tensors at load with heads whose weights are exactly zero, so contiguous sharding puts every dummy on rank 2 and the per-rank geometry (12 q / 2 kv, GQA 6) keeps the R4D attention kernels. Nothing is rewritten on disk. Details in TP3_PADDING_PLAN.md.

TP=3 ./serve-mxfp4.sh     # sets RADIANCE_TP_PAD=3, uses its own cache dir (-tp3pad)
./tp3pad_selftest.py      # check every padding rule against the safetensors headers, no GPU needed
./tp3pad_selftest.py --torch   # also pad one real tensor per rule, inside the image

TP=3 also switches off two production defaults the padded widths cannot serve (RADIANCE_MXFP4_WPERM, RADIANCE_FP8_STREAM), costing ~3-6% decode until the GDN merge gate is relaxed. All-reduces at TP=3 ride RCCL until the 3-rank R4D kernel (ar_oneshot_3rank_exact, libr4d extras rx6) has been measured on three cards; it stays out of the auto-pick list until it passes that hardware gate. Every TP != 3 serve passes RADIANCE_TP_PAD=0 and is byte-identical to before.

KV cache calibration

An explicit --kv-cache-memory beats letting vLLM profile, because profiling is deliberately conservative: it subtracts the profile run's transient activation peak plus the cudagraph estimate, neither of which the steady state needs alongside a full cache. On the R9700 pair that conservatism is 0.93 GiB per rank, 5.7% of the cache: 892,799 KV tokens against 943,581.

That margin depends on the activation peak at CHUNK, on the cudagraph capture set MAXSEQS produces, and on how the allocator fragments on that particular card, so it is measured rather than computed. Measured rows live in kv-profiles.tsv, keyed on hardware and batch shape:

# sig            maxseqs chunk  maxlen  spec    bytes
2x7551-32624     8       8192   262144  dflash  18563072000
1x7551-32624     8       4096   65536   dflash  6535819798

The signature starts with the TP size, so a one-card serve has its own row. That row was measured on the NVFP4 checkpoint, the largest footprint served, so it is conservative for the others. At TP=1 the calibrator inherits the single-GPU profile's CHUNK and MAXLEN defaults so the row it writes is the one the serve will look up, and KV_START=<bytes> starts the search from a pin that is known to serve when the profiling pass itself cannot come up (the NVFP4 case).

A signature that is not in the table falls back to vLLM's own profiling. That is always safe; it just leaves the margin unclaimed, and the launcher prints a one-line note saying so.

To claim it on your own hardware:

./calibrate-kv.sh              # ~15 min, needs the GPUs to itself
./calibrate-kv.sh --dry-run    # show the plan, run nothing
./calibrate-kv.sh --quick      # pass 1 only: pin what vLLM profiles, no search

Pass 1 profiles. Pass 2 raises the pin in 2% steps until the server stops coming up, then backs off one step. A step only counts as passing if the server answers /health and completes a CHUNK-sized prefill plus a decode; reaching GPU KV cache size is not enough, because the cache is allocated before cudagraph capture and capture is where an over-committed pin actually dies.

The result is written to ~/.cache/radiance-mxfp4/kv-profiles.local.tsv, which is read after the shipped table and wins over it. Re-run after changing MAXSEQS or CHUNK: a pin is only valid at the shape it was measured at.

Serving something other than MXFP4

docker-compose.yml serves Qwen3.8-27B-FP8 and is the path for the FP8 and Gemma checkpoints. It is a plain vllm serve with no patch prelude, so it is compose-shaped rather than script-shaped.

./docker-compose-setup.sh --download   # write .env for this host, fetch the checkpoint
docker compose up -d                   # start; follow with: docker compose logs -f
docker compose down                    # stop

Three of its settings are properties of your host and so cannot be committed defaults: the numeric render and video group ids (docker has no --group-add keep-groups, and without them the container sees the device nodes but cannot open them), the HIP indices to serve on, and where the checkpoints live. docker-compose-setup.sh detects all three and writes them to .env, which compose reads on its own — so the file needs no editing. Without a .env, compose stops with the name of the missing variable rather than starting a server that cannot reach a GPU. --show prints what it would write and changes nothing; --download also fetches the checkpoint, using the image's own huggingface_hub so the host needs no Python.

Everything else is ${VAR:-default}, so override it in .env or the shell without editing the file. With podman, podman compose takes the same file. The per-model notes (35B-A3B's --max-num-batched-tokens >= 2240, Gemma-4-31B's template and drafter) are in DOCKERHUB.md.

ParoQuant (int4 W4A8 and int5 W5A8)

MXFP4 is not the only int4 format this stack serves. ParoQuant checkpoints (z-lab/Qwen3.8-27B-PARO: int4 group-128 asymmetric, plus learned pairwise Givens rotations and channel scaling on the activations) run on a W4A8 path built for gfx1201 — hand-written HIP rotation and GEMM kernels, registered as a real vLLM quantization method through the public plugin hook.

The reference ParoQuant implementation is CUDA-only and W4A16. This one is independent: the rotation is a kernel, not a PyTorch fallback, and the asymmetric zero point is folded into the GEMM epilogue as a row-sum correction rather than dequantized into the matmul. Weight-side traffic works out at 4.25 bits/weight, the same as MXFP4.

./setup-paroquant.sh                      # host check, image, checkpoint, drafter, kernels
MODE=prod SPEC=7 ./paroquant/run_paroquant.sh

It shares the image, the DFlash2-FP8 drafter and libr4d with the MXFP4 setup, so running both costs one download of each. Both stacks want both cards and port 8080, so only one serves at a time (vllm-switch paro where the systemd units are installed).

Measured against MXFP4 production on 2 x R9700: GSM8K 500q 97.4-98.0% (MXFP4 97.8), combined decode 226 t/s (MXFP4 186), conc-8 512 t/s, 24.19 ms/step, in-serve numerics gate rel = 0.00000 on every gated shape on both TP ranks (<= 4e-5 at wider M).

int5 (W5A8)

The same path serves 5-bit ParoQuant weights, which is where this stack's fidelity lives. Five bits and not six because the kernels are then unchanged: the int4 GEMM feeds the fp8 WMMA the signed code c - 8, exact in e4m3, and with 5-bit codes c - 16 spans -16..15 where every integer is also exact in e4m3. The GEMM algebra, the zero-point fold, the activation quant, the rotation-stream producers and the split-K / A-tiled bands all carry over; only weight staging changes (low nibbles in the existing word layout, the fifth bit in a byte-per-(slot, lane) plane). int6 breaks that property and would need an int8-WMMA rewrite.

./setup-paroquant.sh --int5                                   # fetch the int5 checkpoint
MODEL_DIR=Qwen3.8-27B-PARO-int5 MODE=prod SPEC=7 ./paroquant/run_paroquant.sh

That is the whole command. The launcher reads the checkpoint's bit width and turns on the activation-quant configuration these numbers were measured with -- int8 per-group activations and the zero-point epilogue -- so there are no RADIANCE_PQ_* flags to remember. Setting any of them explicitly still wins, and the compile-cache directory is keyed on them either way.

Measured on 2 x R9700 (TP=2, fp8 KV, DFlash2-FP8 drafter, SPEC=7) against a same-stack bf16 reference, wikitext, 96 x 500-char chunks:

The bf16 reference has to be served on the same stack. One collected on a different image charges quantization ~0.018 nats that belong to the serving stack's own numerics -- larger than int5's entire quantization KL, and enough to re-rank the builds. Two images' bf16 outputs differ by KL 0.0198 on their own.

Pick int5 for fidelity-sensitive work (logprobs, draft acceptance, long agentic chains where per-token divergence compounds), MXFP4-PARO for maximum prefill and KV headroom. Task accuracy does not separate them: GSM8K is 97.4-97.8% for every variant, inside noise at 500 questions.

MXFP4 weights (W4A8)

A third ParoQuant format keeps the learned rotations on MXFP4 weights (e2m1 + e8m0/32), which puts the GEMM on the zero-VALU fp8-WMMA loop AMD's MXFP4 runs on -- quant_method: paroquant_mxfp4, built by paroquant/build_hybrid.py from the bf16 base and z-lab's rotations, served by paroquant/radiance_paroquant_mxfp4.py. In-serve CHECKALL with real inputs holds rel 0.0012-0.0021 (bf16 rounding) on every shape and partition at TP=2; served-path GSM8K 500q 97.60% (int4 PARO 97.60, AMD MXFP4 97.8); prod decode at parity with int4 PARO (24.53 vs 24.19 ms/step) once the prologue, the stream producers and the merged-linear GEMM each became one launch. See PAROQUANT.md.

The format, the kernels, the knob reference and the rejected experiments are in PAROQUANT.md.

Configuration

Every default below is the measured production configuration, and all of them are ${VAR:-default}, so override from the environment; nothing needs editing. ./serve-mxfp4.sh --help is the short version of these tables, and DRY_RUN=1 ./serve-mxfp4.sh prints the container command a given set of overrides produces without running it.

Paths and runtime

Serving shape

Tool calling and reasoning

The server is started with --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 and this repo's chat template. These are fixed, not EXTRA-tunable.

qwen3_coder and qwen3_xml are the same parser in every image this repo builds or pulls. Both keys are registered and both resolve to Qwen3EngineToolParser, an 8-line shim over vLLM's Streaming Parser Engine:

# vllm/tool_parsers/__init__.py
    "qwen3_coder": ("qwen3_engine_tool_parser", "Qwen3EngineToolParser"),
    "qwen3_xml":   ("qwen3_engine_tool_parser", "Qwen3EngineToolParser"),

They were once two implementations of the same XML grammar (<tool_call><function=name><parameter=k>v</parameter></function></tool_call>, which is what the chat template emits), and on the 0.22-era stack qwen3_coder leaked the closing </tool_call> tag into streaming content — which is why older scripts here pinned qwen3_xml. Upstream deleted both standalone parsers in 2026-06 in favour of the engine, so the distinction is gone; the scripts were standardised on the model card's name, qwen3_coder, on 2026-09-09. Changing the flag between the two names does nothing — if tool markup is leaking, it is not the parser name.

What does matter is patch_qwen3_toolparse.py, applied when the image is built. It fixes streaming-vs-non-streaming divergence on a truncated tool call (vLLM #47137): a clipped opener surfaced <tool_call>\n<function as assistant content in non-streaming while streaming returned None, and a value cut off mid-parameter yielded {} non-streaming against {"city": "San Fr streaming. Both are resolved toward the streaming result — raw tool markup is never surfaced as content. The patch targets the engine, so it applies under either parser name.

Kernels

Diagnostics

RADIANCE_MXFP4_CHECKALL, _SHADOW, _KERNEL_NK, _PERBLOCK_NK, _MHIST, RADIANCE_GDN_NANTRACE and PROFILE_DIR are unset by default and documented where they are read in serve-mxfp4.sh. The full image-level knob reference is in DOCKERHUB.md.

Troubleshooting

First, check three lines in the log

Using RadianceMxfp4W4A8LinearKernel for MXFP4 GEMM     the W4A8 kernel won the selection
[radiance] native MXFP4 enabled on gfx12x              the aiter fp4 gate was relaxed
R4D selections table (RADIANCE_R4D_REPORT=1, on)       which kernels bound, and why not

A kernel that fails to bind falls back silently and costs performance rather than raising, so read that table rather than assuming.

Symptoms

The launcher preflights the host and fails with the command that fixes it, so most problems surface as a one-line error rather than a traceback.

Benign log lines

Three lines come up often enough to be worth naming. None of them is a problem.

INFO ... [weight_utils.py:890] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) — informational, and the good case. vLLM only prefetches checkpoint shards into page cache when the weights sit on a network filesystem; on a local disk the mmap read is already optimal, and --safetensors-load-strategy=prefetch forces work that buys nothing.

Loading safetensors checkpoint shards: 80% Completed | 0/1 — cosmetic. There are two progress bars (the target's shards, then the DFlash2 drafter's single shard), and vLLM deliberately uses a newline-terminated bar format rather than a redrawing one so it stays readable under multiprocessing (weight_utils.py), so the per-line log prefixer can stitch a fragment of one bar onto the other. n/total is the authoritative field, not the percentage — 100% Completed | 1/1 follows.

WARNING ... [rocm.py] Using CUDA_VISIBLE_DEVICES on ROCm is deprecated — nothing sets that variable: on ParoQuant builds vLLM mirrors our HIP_VISIBLE_DEVICES into it at import and then warns about its own copy. Harmless in the server log; it was only load-bearing in the cpuinfo subprocess above.

Performance

Measured with BetterBench 0.4.0, corpus v1.0, 2026-08-30, on the reference box: 2x R9700, TP2, dflash SPEC=7, GPU_UTIL=0.98, KV pinned, temp 0.7 / top_p 0.95 / top_k 20, cold prefix cache per request.

Two notes on reading these numbers:

• Report step time, not tokens/s, for anything decode-related. Tokens/s swings ~14% on draft-acceptance luck alone at a fixed config; ms/step = 1000 x (accepted/draft + 1) / tok_s divides that out.
• Speculative decoding packs several tokens into one stream update, so there is no per-token latency to quote. update p50 is the wall-clock gap between updates, and tok/update is how many tokens land in each.

Single stream

One pass per category, so treat each cell as a single observation rather than a distribution. The update p50 column is the exception worth trusting: it is the median of hundreds of update gaps inside one run, and it lands within 0.2-0.5 ms of the 22.66 ms/step figure the decode work was gated on.

Combined (weighted code 0.3, reasoning 0.2, prose 0.15, json 0.15, file_edit 0.1, summarization 0.1): decode 224.3 t/s, update p99 23.4 ms, TTFT p50 53 ms.

The spread across categories is almost entirely tok/update, not step time. Every category holds 22.8-23.2 ms per update, and code reaches 253.8 t/s against prose's 109.4 purely because code drafts accept 5.77 tokens per update where prose accepts 2.54. That is why the depth default is content-dependent; see SPEC in Serving shape.

Concurrency

48 requests per level, MAXSEQS=8.

Aggregate throughput peaks at 8 and does not improve at 16, because MAXSEQS is 8: the extra requests queue, which is what the 4430 ms TTFT is. Serving 16 concurrent usefully means MAXSEQS=16, which also widens the decode band to RADIANCE_MXFP4_DECODE_MAX_M=128; measured there, conc-16 reaches 549-622 t/s at 75-79 ms steps. That path re-profiles KV rather than using the pin, so run ./calibrate-kv.sh if you standardise on it.

Prefill

8 runs per depth, cold prefix cache.

This sweep stops at 64k. For deeper context the reference points are the fp8-attention gate (3831 t/s at 106k) and the 0.5.8 -> 0.7.4 table, both in PERFORMANCE.md; they run on a different harness and are not directly comparable to this table. Prefill throughput keeps falling with depth in both.

One card (TP=1)

BetterBench --quick (5 passes per category, 8 per prefill depth, 48 requests per concurrency level), 2026-09-16, one R9700 at the box's standing 210 W cap and -75 mV, ./serve-tp1.sh defaults (MAXSEQS=8, 65,536 context; lazy GDN snapshots are OFF), native MXFP4 checkpoint.

Combined: decode 137.7 t/s, update p99 35.7 ms, TTFT p50 95 ms. Before the 2026-09-16 pass the same card served at 64 ms per update (75.4 t/s combined).

Prefill is compute-bound and follows the core clock: the same config measured 3552 t/s at 2k and 3192 at 64k with the cap raised to 300 W (undervolt kept), and concurrency-8 rose to 471 t/s. Single-stream decode went the other way there (42 ms per update whenever the core boosted past 3.3 GHz), so the power policy is a real trade on one card; the reference box's two cards also differ by ~12% on prefill under the same cap, so compare prefill on the same card only. GSM8K 250q greedy on this config: 97.6-98.4% across the runs of the day, the same band as two cards.

Quality and capacity

The change-by-change ledger (what each landed optimization was worth and how it was gated), the 0.5.8 -> 0.7.4 provenance A/B, and the gated-delta-net NaN write-up all live in PERFORMANCE.md.

How the MXFP4 path works

Native MXFP4

Quark OCP micro-scaling checkpoints (quantization_config.quant_method: quark, mxfp4 weights and activations, group 32, e8m0 scales) run natively with RADIANCE_MXFP4=1. Drop --quantization: the runtime reads the method from config.json and routes it itself.

Without this, vLLM falls back to emulated MXFP4, which materialises every weight tensor in bf16 on each forward. Nothing in the way was a compiler limitation, since Triton 3.6 does lower tl.dot_scaled on gfx1201. It was three soft gates, all handled in patch_quark_mxfp4.py: an is_fp4_avail() allowlist that omits gfx1201, an aiter module path that moved in 0.1.17, and gfx1250 tiles that ask for matrix_instr_nonkdim=32 when this card's WMMA is 16x16x16 only (mxfp4-configs/ pins 16 across every band).

The native path is bit-identical to emulation (the activation quantization is the same either way), so it is a speed change with no quality dimension. Measured on gate_up 17408x5120: 6.1x at M=16, 4.7x at M=32, 2.5x at M=64.

The stock emulated path cannot serve these checkpoints on this card at all, which makes the native kernel the only way to run them here, not merely the faster one. (Quark's TileLang backend cannot initialise inside a vLLM worker, and the retired RADIANCE_MXFP4_MAX_M fallback was specialised into the compile graph during the profile run, so it killed startup rather than one request.)

The W4A8 fp8-WMMA kernel

RADIANCE_MXFP4_W4A8=1 additionally routes linears to a hand-written fp8-WMMA HIP kernel (radiance_mxfp4_fp8.hip). Triton lowers tl.dot_scaled by upconverting e2m1 to bf16 and using the 16-bit WMMA; register-resident on this card, fp8 WMMA runs 325 TFLOP/s against f16's 160, while Triton's own fp8 tl.dot manages 43 because it upconverts and pays conversion on top. Against the tuned aiter path it measures 1.47-2.26x faster and 4.2x more accurate (0.0265 vs 0.1119 relative error), since fp8 activations beat the mxfp4 ones aiter quantizes to.

It is off in the image because it changes numerics (the layer becomes W4A8 rather than the checkpoint's declared W4A4: more precise than what the model was calibrated against, but no longer bit-identical), and on in serve-mxfp4.sh, which is what every number above was measured with.

There are two tilings, because prefill and decode are different problems.

At decode M is 5 (batch 1 x num_speculative_tokens+1), where the prefill tile issues 51x more matrix MACs than useful: 4352 WMMA per wave against 5 real rows. The decode split-K reduction is fused, so the KS blocks covering one output range race on an atomic counter and the last arrival reduces in place with no second launch, worth a further -3.9% of step time at bit-identical output. BK=128 winning at decode reverses the prefill answer because that -34% was purely the LDS occupancy cliff, which a 16-row A tile never reaches.

RADIANCE_MXFP4_DECODE_MAX_M must cover MAXSEQS x (SPEC+1) rows or the biggest verify batches fall back onto the prefill tile. RADIANCE_MXFP4_W4A8_MIN_M defaults to 0, so our kernel serves every M.

4-bit weights leave far more room for KV. On 2x R9700 the 27B MXFP4 body occupies 9.24 GiB/GPU against roughly 12.6 for the same model in FP8, and that headroom goes straight into context.

NVFP4 checkpoints: online requantization to MXFP4

RADIANCE_NVFP4_MXFP4=1 serves compressed-tensors NVFP4 checkpoints (e.g. unsloth/Qwen3.8-27B-NVFP4: e2m1 weights, e4m3 scale per 16, fp32 scale per tensor) on the same W4A8 fp8-WMMA kernel as the Quark MXFP4 release, the way AMD's ROCm blog does it for CDNA4 inside SGLang: each NVFP4 linear is loaded as stored, then dequantized and requantized to e2m1 + e8m0/32 in process_weights_after_loading (radiance_nvfp4.py, hooked by patch_nvfp4_mxfp4.py), per partition of a merged linear so gate_up_proj's two global scales are honoured instead of collapsed. The block exponent is chosen per 32-block by squared error between the no-clip rule and one binade finer (RADIANCE_NVFP4_EXP=mse, default; ocp and noclip are the fixed rules). Conversion takes ~10 s per rank for the whole model.

The step is lossy: measured on the real tensors, NVFP4 sits 0.113 relRMS from the bf16 original (19 dB SQNR) and the requantized MXFP4 0.158 (16 dB) -- a direct bf16 -> MXFP4 quantization would be 0.112, so the cost is the double rounding, not the format. GSM8K does not see it (below).

That checkpoint is mixed precision: only the first 56 layers' MLPs are NVFP4; attention, the GDN projections, lm_head and the last 8 MLPs are FP8 per-channel. RADIANCE_NVFP4_FP8_LAYERS=mxfp4 (default) requantizes those too (never lm_head), which puts every linear on the radiance kernel and lets the fp8 residual stream and GDN norm+quant epilogues attach; GSM8K 500q scored 97.40%, the PARO-MXFP4 production number. fp8 leaves them on vLLM's FP8 path (torch._scaled_mm, a real hipBLASLt fp8 GEMM on gfx1201, unfused scaling, no radiance fusions) and is diagnostic only: it wedged the GPU (driver reset) twice under sustained 8-way concurrency, minutes into GSM8K. The int2 draft head and verify head read the FP8 lm_head directly (e4m3 rows decoded in the rerank kernel). RADIANCE_NVFP4_BF16_LAYERS=in_proj_ba (default) also requantizes the bf16 GDN a/b gate projections so the GDN in_proj merge fuses all 48 layers. Measured on 2x R9700 against the PARO-MXFP4 production unit the same day: GSM8K 500q 97.60% (vs 97.00 / 97.60 record), decode 22.2 / 23.8 / 24.7 ms/step at 25 / 8k / 32k context (vs 23.3 / 24.8 / 25.6), prefill 5072 / 4845 / 3900 tok/s at 8k / 26k / 104k (vs 4871 / 4679 / 3751), BetterBench quick conc-8 aggregate 597 t/s (vs 512). To serve it:

RADIANCE_NVFP4_MXFP4=1 SNAP=~/models/Qwen3.8-27B-NVFP4 NAME=vllmnvfp4 KV_MEM=0 GPU_UTIL=0.95 \
  CACHE=~/.radiance-cache-nvfp4-093 SERVED_NAMES="Qwen3.8-NVFP4 Qwen3.8 Qwen3.6" ./serve-mxfp4.sh

Why the drafter is FP8, not MXFP4

The target checkpoint is Qwen3.8-27B-MXFP4-mtpfp8: AMD's Qwen3.8-27B-Quark-AWQ-MXFP4 body with the MTP drafter requantized to fp8 by fp8_mtp.py.

Four-bit costs more acceptance than it saves in bandwidth, and AWQ does not rescue it. Data-free RTN measured ~11.6% relative error and cost acceptance 2.5 -> 2.21; AWQ calibration improved that by 0-5% (the alpha search chose 0.1-0.2, and 0.0 for mtp.fc, because MXFP4's per-32 E8M0 block exponent already does most of what per-channel scaling would). FP8 e4m3 per-channel is ~2-3% relative error and holds acceptance at 2.60-2.80, removing ~17% of decode weight traffic instead of 25%: the smaller win that actually holds.

MXFP4-NOTES.md collects the longer form of this reasoning, and serve-mxfp4.sh is the worked launch, with the measurement that chose each default.

What's inside the image

Everything below is baked into the image, and the tuned paths are env-gated and on by default. See DOCKERHUB.md for the per-knob reference: every flag, its default, and what it does.

Building the image from source

You do not need any of this to serve. setup-mxfp4.sh / serve-mxfp4.sh point at IMAGE=ggz14/vllm-radiance-mxfp4:latest, which the recipes below build in a few minutes. Build only to change a pinned component or a baked-in patch.

Three recipes, all with the repo as flat build context (below, the Dockerfile one):

./build.sh            # .top on the published base -> $VERSION-$SHA, latest, v$VERSION (minutes)
./build.sh --paro    # the ParoQuant image, same flow
./build.sh --full    # the five-stage release build (hours: the four from-source stages)
./build.sh --base-only  # the bare base, to publish and re-pin the variants
./build.sh --push    # push the image afterwards
./build.sh --no-cache  # force a recipe build even though nothing moved

build.sh pulls the published default and passes its live digest as --build-arg BASE_DIGEST (a base repaint cannot slip in); the version string lives in VERSION. podman build takes the same arguments; a bare docker build -f Dockerfile.ggz14.top .-style run works too, but uses the recipe's baked-in BASE_DIGEST default instead of build.sh's live pin.

No GPU required to build, no model on the build machine either: hipcc compiles for gfx1201 without a device (the same mechanism as the kernels and the pre-baked AITER module), and each recipe's gates import every radiance module in the venv before the image is declared built, so a broken change fails the build rather than the first container start.

The Dockerfile recipe builds everything from source in four stages:

Expect it to run for hours on a many-core box; it is a full PyTorch compile.

Why the prune matters. The stock ROCm base is 7.4 GiB compressed on its own, most of it device code for GPUs this image cannot run on. Pruning to gfx1201 and shipping an allowlist takes the image from 9.35 GiB compressed to 3.66 GiB. The prune must happen in a stage the release stage copies from: deleting files in a layer stacked on the base reclaims nothing.

The release stage still ships a working compiler (hipcc, g++, and the C++/Python headers). That is not slack to trim: AITER JIT-compiles its kernels on first use inside the running container, so an image without those headers boots and then dies on the first AITER module build. The build asserts it by compiling and importing a pybind11 HIP module.

Component pins are the ARGs at the top of the Dockerfile (TORCH_VERSION, TRITON_VERSION, TORCHVISION_VERSION, AITER_VERSION, VLLM_VERSION). Each is both the git tag that gets compiled and the version the resulting wheel reports, and the build asserts the two agree, so pip show and the startup banner can be trusted.

Do not bump torch / triton / torchvision on their own. They are not independent choices: vLLM pins the torch version it is tested against, torch pins its triton, and torchvision ships a matching release. The build runs vLLM's own use_existing_torch.py, which strips those pins, but that exists so pip does not re-download torch, not as licence to install a newer one. Builds 0.5.0 through 0.5.4 compiled against a newer trio and hung a GPU under sustained tensor-parallel load; restoring the pinned versions fixed it with no code change. If you override these with --build-arg, move them together and soak-test under real load.

If you just want a known-good image without building, setup-mxfp4.sh already points at ggz14/vllm-radiance-mxfp4 (the MXFP4 image). stilldeadcode/vllm-radiance is the previous stock-generation image -- the bare base that builds its kernels in the container at first start -- kept for reference and as the published pin for the top recipes.

Repository layout

Flat build context. The runtime Python modules (radiance_*.py), the patch_*.py fixes, the fp8-configs/ moe-configs/ and mxfp4-configs/ GEMM configs, the chat templates, and all three Dockerfiles (Dockerfile, Dockerfile.ggz14.top, paroquant/Dockerfile -- build.sh drives them) live at the repo root so docker build -f <recipe> . works directly. prune_rocm.sh is the ROCm slimming step, and it self-checks: the arch's own kernels must survive and hipcc must still link a HIP shared object, since AITER JITs at runtime (for the modules the recipes do not pre-bake).

MXFP4 entry points

ParoQuant entry points

Where the HIP kernels live

The HIP kernels are not in this repo. They live in libr4d, a library of kernels for gfx1201 rather than for any one model, pinned by tag (R4D_VERSION in the Dockerfile), which the build asserts against the version the compiled library reports. make r4d clones and builds that pinned tag here for development, so a locally built r4d.so matches the one in the image.

R4D entry points are named for the geometry they are compiled for (attn_decode_h256_gqa6_fp8kv, gdn_chunk_scan_k128_v128_c64_bf16, ar_oneshot_2rank_exact) and reject a mismatch rather than running, so which kernels an engine binds is visible in the startup log and in r4d.kernels().

One HIP kernel does stay here: radiance_mxfp4_fp8.hip, the MXFP4 W4A8 fp8-WMMA GEMM. It is specific to this fork rather than general to gfx1201, so the image build compiles it directly and make radiance_mxfp4_fp8.so rebuilds it against the image toolchain during development.

Further reading