English | 繁體中文

WHIRL

Windows HIP Inference for RDNA LLMs

WHIRL is an LLM inference engine for the AMD Radeon AI PRO R9700 (RDNA 4) on Windows.

Design principles

1. Built for AMD GPUs on Windows. A native Windows C++/HIP inference engine with hand-written, hand-tuned kernels for RDNA. No WSL, no Docker, no Linux VM: install the AMD driver and run.
2. Fully open, kernels included. Everything from host-side scheduling to every GPU kernel is open source under Apache-2.0, with no closed binary components. Bug reports, benchmark results and code contributions are all welcome. Let's make AMD inference on Windows as good as it can be.
3. Accuracy before speed. Every speedup (speculative decoding, batching, prefix caching, cache restore) produces output bit-identical to plain greedy decoding, enforced by standing tests. WHIRL never trades away accuracy beyond the model you chose.
4. Few models, pushed to the limit. WHIRL doesn't try to run everything. It targets specific models on specific hardware and pushes them close to the hardware limit, adding architectures one at a time.
5. Transparent, verifiable numbers. Every figure is measured, with methods, commands and raw data published. For content-dependent features such as speculative decoding, we report the best case, the typical case and the worst case, so you know what to expect.
6. Designed for real workflows. Coding agents and long contexts come first: fast long-prompt processing, a responsive main conversation while others share the GPU, and a VRAM → RAM → SSD cache that survives server restarts.

The architectures supported today are qwen35 (dense) and qwen35moe (mixture of experts); more will be added one by one. WHIRL follows the design philosophy of NInfer (Apache-2.0) — build for selected models and selected GPUs, as close to the hardware limit as possible; zynfer carried the same idea to RDNA 4.

WHIRL vs llama.cpp at a glance

Same GGUF file, same prompts, one R9700, greedy decoding; llama.cpp b11214 (ROCm) with the fastest flags we found for each row. Every WHIRL speedup gives output bit-identical to plain greedy decoding (why).

¹ KV cache: WHIRL int8 (q8h) / llama.cpp f16; llama.cpp decodes without speculative decoding at 256k.

Full table (including Qwen3.8 Q4_K_M and where WHIRL does not lead by much): Performance.

Speed without changing the answer

Every acceleration in WHIRL is checked to produce the same tokens, bit for bit, as plain greedy decoding of the same model file:

What WHIRL does not do to go faster: no KV cache below 8 bits (no 4-bit KV), no requantizing the model's weights to fewer bits (only the draft-only MTP and draft-head copies, which cannot change the output), no fp8 attention.

To be precise about what this means:

• Weights are whatever GGUF you choose; their quantization is the file author's choice, and WHIRL computes with it as stored.
• KV cache defaults to f16. On the dense models, when f16 does not leave room for a long context, the server falls back to int8: q8v (f16 keys, int8 values), then q8h (int8 keys and values); the MoE model stays f16. WHIRL_KV pins the format. Both are numerically lossy compared with an unquantized model, like any quantized inference. WHIRL does not claim "zero loss".
• Continuing a conversation reuses the KV and DeltaNet state computed for the previous turn, including the tokens generated by decoding. They come from the decode kernels (and from a different split of the prompt into chunks), so they are numerically equivalent but not bit-identical to re-processing the whole history as one prompt; the difference is documented and measured (kv-and-caching.md).
• The MXFP4 files run in speed mode: prefill uses fp8 activations in the MXFP4 matrix multiplies and f16 in parts of the DeltaNet prefill. Decode and verification always use the same int8 path in both modes, so the guarantees above hold in both.

Requirements

Our test machine connects the R9700 as a USB4 / Thunderbolt eGPU; that works. Numbers on a card in a direct PCIe slot may differ slightly (mostly where data crosses the host link, such as model loading and the RAM / SSD tiers).

Not supported yet: other AMD GPUs. The RX 9070 series has the same chip (gfx1201) but 16 GB, most likely too little for these 27B / 35B models (untested). Radeon 8060S (Ryzen AI Max+ 395, gfx1151) support is in development on the gfx1151 branch.

Quick start

1. Download whirl-0.1.3-windows-x64.zip from Releases and a supported model.
2. In PowerShell:

Unblock-File .\whirl-0.1.3-windows-x64.zip
Expand-Archive .\whirl-0.1.3-windows-x64.zip -DestinationPath C:\whirl
cd C:\whirl\whirl-0.1.3-windows-x64
.\whirl.exe devices                                    # should list AMD Radeon AI PRO R9700 (gfx1201)
.\whirl.exe chat C:\models\Swift-1.5-Qwen3.8-27B-MXFP4-A-outQ6_K.gguf "Explain TCP slow start in two sentences." --max-tokens 400
.\whirl-server.exe C:\models\Swift-1.5-Qwen3.8-27B-MXFP4-A-outQ6_K.gguf --port 8080

The server speaks the OpenAI API at http://127.0.0.1:8080/v1 (/v1/chat/completions, /v1/completions, /v1/models, /health), so any OpenAI-compatible client works. Stop it with Ctrl+C; cached sessions are written to the SSD tier first.

For long documents, give a slot a longer context, for example --ctx-per-slot 262144 (256k tokens per request); see recipes.md.

The first run with a model tunes the GPU kernels once — about 1.5 minutes for a 27B Q4_K_M file, a few seconds for MXFP4 files — and caches the result in %LOCALAPPDATA%\whirl. Full walkthrough: quick start. Every option: usage reference.

Supported models

WHIRL reads general.architecture from the GGUF and accepts only qwen35 and qwen35moe (the Gated DeltaNet + gated attention hybrid, with an optional MTP head) and fine-tunes that keep that architecture. Other architectures (Llama, Gemma, Mistral, DeepSeek, qwen3, qwen2, …) are refused at load time with a clear message, and so are tensor types WHIRL has no kernels for (for example NVFP4); there is no slow generic fallback.

Recommended: our own MXFP4 quantizations give the best performance.

• Dense 27B: Swift-1.5-Qwen3.8-27b-MXFP4-GGUF, variant A (Swift-1.5-Qwen3.8-27B-MXFP4-A-outQ6_K.gguf). Swift-1.5 also thinks much less than the base model (44% fewer thinking tokens on our mixed Chinese/English coding prompts).
• MoE 35B (~3B active): Ornith-1.5-35B-A3B MXFP4: tsaipifong/Ornith-1.5-35B-A3B-MXFP4-GGUF.

Tested GGUF files (loaded, greedy output checked, MTP output identical to plain greedy, benchmarked on the R9700):

Image input works with a Qwen3-VL-style mmproj (F16 / BF16) via --mmproj; it was tested with the 27B files above. Other qwen35 / qwen35moe GGUFs should load but are untested. Choosing and making GGUFs: quantization.md.

Performance

WHIRL 0.1.3 vs llama.cpp b11214 (ROCm, the fastest flags we found for each row) on the R9700, greedy decoding, the same GGUF files and the same prompts in both engines. Decode numbers lead with WHIRL's default mode, MTP + n-gram; rows marked no MTP compare plain decoding without speculation. The KV cache format is shown per row.

Our MXFP4 files (Swift-1.5 27B and Ornith-1.5-35B-A3B) are the main comparison; Qwen3.8-27B UD-Q4_K_M is shown for reference only. From 96k tokens on, each cell gives the KV format each engine actually used, as KV WHIRL / llama.cpp.

¹ llama.cpp's best of plain / MTP / MTP + n-gram for that row.

² llama.cpp has no automatic persistent KV cache.

³ llama-bench tg64@d262144: 64 tokens generated after a 262,144-token context, plain decoding without speculative decoding. llama.cpp ran Swift-1.5 MXFP4 at 256k with f16 KV (it fits, 31.3 GB); Ornith-1.5 at 96k and 256k also used f16 KV; Qwen3.8 Q4_K_M at 256k needed q8_0 KV, because f16 spilled into shared memory.

⁴ llama.cpp re-measured for v0.1.3 with the same prompts as WHIRL; the mode used (the fastest measured): decode after 16k: Swift-1.5 MTP + n-gram, Ornith-1.5 plain, Qwen3.8 MTP + n-gram; 4 concurrent users: Swift-1.5 plain, Ornith-1.5 plain, Qwen3.8 plain; 26k system prompt TTFT: plain only (a 1-token reply; speculative decoding does not apply).

Speculative decoding: best, typical and worst case — Swift-1.5 27B MXFP4-A, WHIRL's default MTP + n-gram against decoding without speculation; the output is the same bit for bit:

Long context with int8 KV (q8h) — WHIRL server, one user, --ctx-per-slot 262144, q8h KV (int8 keys and values; queries and keys Hadamard-rotated first), the setting used for 256k tokens on the dense models (set with WHIRL_KV=q8h for the MoE model):

Where WHIRL does not lead by much: plain decoding (no MTP) of the dense models is memory-bandwidth bound in both engines (1.13–1.14× llama.cpp); decode after a 16k-token context leads by only 1.17× (Swift-1.5) and 1.33× (Qwen3.8) on the dense models; prefill of Qwen3.8 Q4_K_M leads by less (1.51× at 96k and 1.44× at 256k tokens, against 2.28× and 1.74× for Swift-1.5 MXFP4); Ornith-1.5 prefill at 256k leads by 1.48×; and in the worst case for speculative decoding (new writing after 128k tokens, nothing to copy) it gains only 1.56×. Our R9700 is a USB4 eGPU; prefill and decode stay in VRAM, but restores and model loading cross the host link.

Full methodology and all numbers: benchmarks.md.

What's new since v0.1.0

Both versions measured on the same machine, with the same GGUF file and the same prompts; the greedy output of the two versions is bit-identical.

v0.1.2 for reference (same machine, same output): 53.5 tok/s for the 127.9k request, 65.0 tok/s for the session; the gain is v0.1.3's.

Windows security prompts

whirl.exe and whirl-server.exe are not code-signed. Windows SmartScreen may show "Windows protected your PC" for a freshly downloaded copy: click More info → Run anyway, or run Unblock-File on the zip before extracting it (only for a zip from the official releases page, after checking its SHA-256). With Smart App Control set to On, Windows may block the programs without a "Run anyway" option. Details: windows_security.md.

Documentation

Development and AI assistance

WHIRL was designed, implemented, optimized and benchmarked with the participation of Claude Opus 5.5, an AI model by Anthropic, working under the direction of the project owner (@tsaipifong). Anthropic is not affiliated with and does not endorse this project.

License and acknowledgements

Apache License 2.0; see LICENSE and NOTICE. Third-party material is listed in THIRD_PARTY_NOTICES.md, the origin of every source file in PROVENANCE.md.

• NInfer — the design philosophy WHIRL follows (ideas only).
• llama.cpp / ggml — used as the comparison baseline in our benchmarks and as the reference for GGUF and quantization-format compatibility; the few adapted pieces (tokenizer pre-split / BPE loop, lookup tables, image preprocessing) are MIT-licensed and listed in THIRD_PARTY_NOTICES.md.
• gufo and r9700-stack — measurement methodology and optimization ideas.
• Model authors: the Qwen team (Qwen3.8), unsloth, FreedomAISVR, ornith-ai (Ornith-1.5), and ukisai (Swift-1.5, the fine-tune our MXFP4 files are quantized from).