vllm-rocm-windows-RDNA2-oneclick

中文版文档：README_zh-CN.md

Native vLLM + ROCm 7.15 (TheRock) for the whole AMD Radeon RDNA2 family on Windows — no WSL2, no NVIDIA, no compiler. One-click installer, everything prebuilt, and an OpenAI-compatible chat server that looks and works like the NVIDIA stack. Code and build pipeline are ready for RDNA3 (RX 7000) and RDNA4 (RX 9000) too — those families are experimental until hardware validation (see docs/MULTIARCH.md).

Verified on AMD Radeon RX 6750 XT 12 GB (gfx1031) — Windows 11 native — August 2026 RX 6800 (gfx1030): testing tracked in issue #2 — confirmation pending

⭐ If it worked, please leave a star, thank you! ⭐

GPU support matrix

INSTALL.bat auto-detects the family; INSTALL.bat -Variant rdna3 forces one.

---

Verification — real output from the RX 6750 XT

1. ROCm detects the GPU (native Windows process, no WSL)

2. Raw GPU power — native rocblas-bench.exe, FP16 GEMM 4096³

3. vLLM tuned decode

---

Requirements

No compiler, no ROCm installer, no manual setup — everything ships prebuilt.

Quick start (one-click)

1. Download the repository (ZIP or git clone). You do not need to download the release archives (*.tar.zst) manually — the installer fetches them automatically from the Releases tab.

2. Double-click INSTALL.bat — by default it downloads from this repo's releases; pass a GitHub username to use your own fork instead, -Variant to force a GPU family. It checks GPU + disk, then installs everything it does not already have (re-run is always safe and fast):

   Step	Action
   1/6	GPU family detection (rdna2/rdna3/rdna4, warns on unknown GPUs) + Python 3.11.9
   2/6	4 archives from GitHub Releases → C:\Python311, C:\TheRock, C:\vw_*_build
   3/6	venv fix + torch self-check
   4/6	Qwen3.5-4B 4-bit model (skipped if already in HuggingFace cache)
   5/6	writes config.bat (the file to edit to change model later)
   6/6	verification benchmark

3. CHAT.bat → opens the web chat in your browser (starts the model server automatically the first time). By default the model answers directly (THINKING=0 in config.bat); set THINKING=1 to let the model explain its reasoning — with Qwen3.5 the reasoning text then appears inside the answer (this template does not split it into a separate field). Tokens stream live with a tok/s counter, everything local on your AMD GPU.

4. SERVE.bat → starts the model server on its own (OpenAI-compatible API on http://127.0.0.1:8000/v1, like vllm serve on NVIDIA). Use it with any OpenAI client, or just run CHAT.bat.

5. VERIFY.bat → all 3 verification checks in one run: ROCm GPU detection, native rocBLAS FP16 power (~26 TFLOPS) and the full 512-token vLLM benchmark (~58-62 tok/s).

6. UNINSTALL.bat → clean removal of the whole stack when you no longer need it (keeps the repo folder itself).

Benchmarks

Native ROCm vs Vulkan (llama.cpp)

The only other way to run LLMs on RX 6000 under Windows is the llama.cpp Vulkan backend (AMD never shipped ROCm for these cards on Windows). Community measurements consistently put Vulkan at 70-80% of native HIP throughput on the same RDNA2 silicon, and Vulkan cannot run vLLM at all (OpenAI-compatible serving, continuous batching, PagedAttention) — that is what this stack adds:

*Estimated from the commonly reported 70-80% of HIP throughput; measured on llama.cpp HIP vs Vulkan on RDNA3 (70-80%) and RDNA2 (same range) by craftrigs.com.

Full optimization history:

How it works

1. TheRock builds ROCm (HIP runtime, rocBLAS, Tensile) as native Windows binaries — this is what makes ROCm exist on Windows at all.
2. HSA_OVERRIDE_GFX_VERSION=10.3.1 presents any RDNA2 GPU (gfx1030/1031/1032) as gfx1031 — the arch the packaged rocBLAS/Tensile libraries and torch_gfx1031.kpack were built for; the HIP kernel ships as a fat binary (gfx1030 + gfx1031 + gfx1032) so the whole RX 6000 series runs native code.
3. PyTorch 2.12 +rocm7.15 links against that runtime → torch.cuda.is_available() == True on RDNA2 Windows.
4. vLLM plugin vllm_windows_rocm registers the tuned kernels: native HIP W4 GEMV for quantized linears, M=1 skinny GEMV for dense ones (including the huge tied lm_head), CUDA-graph safe (registered as real torch ops).
5. INSTALL.bat (engine: INSTALL.ps1, manifest: MANIFEST.json) downloads the 4 prebuilt archives from GitHub Releases and the model from HuggingFace, installs base Python, fixes the venv, verifies with a benchmark. Idempotent: it only downloads what is missing.

Integrity (SHA256)

Every downloaded part is verified against SHA256SUMS.txt published on the same GitHub release, before extraction:

• a corrupted or truncated download is rejected with checksum mismatch — re-run INSTALL.bat, it resumes automatically
• a failed download never leaves a partial file behind (it is deleted and retried once)
• forks/mirrors without the checksum file: pass -SkipChecksum

scripts\make_checksums.ps1 regenerates the checksum file; scripts\validate_release.ps1 checks the manifest against the release (both run in CI). See docs/RELEASING.md and SECURITY.md.

Installed layout

C:\Python311                                 Python 3.11.9
C:\TheRock\.venv                             torch 2.12+rocm7.15 venv (vLLM 0.19.1)
C:\TheRock\build\dist\rocm                   ROCm runtime libraries
C:\TheRock\ROCM_VLLM_RUNTIME                 vLLM + plugin + rocBLAS + rocblas-bench
C:\vw_cext_build, C:\vw_hipgemv_build        native HIP kernels
%USERPROFILE%\.cache\huggingface             model weights

Release archives (this repo's Releases tab, tag V2.1):

Repository structure

vllm-rocm-windows-rdna2-oneclick/
├── INSTALL.bat               one-click installer (entry point)
├── INSTALL.ps1               installer engine (downloads, verifies SHA256, extracts)
├── UNINSTALL.bat             clean removal of the installed stack
├── UNINSTALL.ps1             uninstaller engine
├── CHAT.bat                  opens the web chat (auto-starts the server)
├── SERVE.bat                 starts the model server alone (OpenAI API)
├── VERIFY.bat                all 3 verification checks in one run
├── chat.html                 AI-Studio-style chat UI (Thinking spinner + tok/s + run settings)
├── MANIFEST.json             release asset manifest, one variant per GPU family
├── SECURITY.md               vulnerability reporting + supply-chain notes
├── CONTRIBUTING.md           how to verify GPUs, report bugs, contribute
├── plugin_overrides/         tuned plugin modules (awq_gemv, bf16_gemv)
├── kernels/                  native HIP W4 GEMV source + per-family build script
├── scripts/                  model server, benchmark, checksums, release validation
├── docs/                     multi-arch + release guides
├── assets/                   verification screenshots
└── results/                  raw benchmark logs + progression chart

Using a different model

Edit config.bat (written by the installer): set SERVED_MODEL to the model folder and MODEL_NAME to the name shown in the chat / used by the API, then run CHAT.bat again. The benchmark uses BENCH_MODEL (same file):

C:\TheRock\.venv\Scripts\python.exe -c "from huggingface_hub import snapshot_download; print(snapshot_download('<owner>/<repo>'))"

prints the snapshot folder to put in config.bat.

Pick the model at install time

INSTALL.bat accepts a model id, so you never need to edit config.bat:

INSTALL.bat                        rem default: cyankiwi/Qwen3.5-4B-AWQ-4bit (Qwen/Qwen2.5-1.5B-Instruct-AWQ on 4 GB cards, detected automatically)
INSTALL.bat -Model <owner>/<model> rem any public AWQ/GPTQ model
INSTALL.bat someuser               rem from YOUR fork's releases
INSTALL.bat -Variant rdna3         rem force a GPU family (rdna2/rdna3/rdna4)

or, with the engine directly (any public AWQ/GPTQ model):

powershell -ExecutionPolicy Bypass -File INSTALL.ps1 -Model <owner>/<model>

Recommended models for 12 GB VRAM (AWQ 4-bit)

*Same RX 6750 XT. Larger context windows reduce tok/s proportionally. Note: the Qwen3.5 8B-class model is the 9B dense — no AWQ/GPTQ quant is published for it yet (only MLX/GGUF), so any public AWQ 4-bit model works here; the stack is not Qwen-specific.

Troubleshooting

Rebuild the kernel

kernels\build_kernels.ps1 recompiles kernels/src/gemv_w4.cu per GPU family (fat binary for gfx1030+1031+1032, or gfx1100/1200 for RDNA3/4; needs HIP SDK

• MSVC). Only for non-RDNA2 targets or exotic torch ABIs; the shipped rdna2 fat binary covers all RDNA2.

Acknowledgments

Built on ROCm/TheRock, PyTorch ROCm and the vLLM project. Sincere thanks to MrPie (ThePie88) for the upstream awq_gemv / bf16_gemv / gemv_w4 kernels included (and credited) in plugin_overrides/ — Apache 2.0, see THIRD_PARTY_NOTICES.md. Not affiliated with AMD.

License

Apache License 2.0 — third-party components and trademarks are covered in THIRD_PARTY_NOTICES.md.