vLLM for AMD on Native Windows

[!IMPORTANT] Thank you, AMD

AMD's Threadripper platform and Radeon AI PRO R9700 hardware made this native-Windows ROCm project possible.

Run the real vLLM engine directly on an AMD GPU in native Windows: no WSL, Linux VM, Docker container, CUDA translation layer, or GGUF plugin required.

This is an experimental, community-maintained fork of vLLM, currently based on vLLM v0.28.0. It is validated on Windows 11 with a Radeon AI PRO R9700 (RDNA4, gfx1201) and AMD's Windows ROCm/PyTorch wheels. It provides an OpenAI-compatible server, offline benchmarks, FP8/MXFP4 support where vLLM has a compatible kernel, and a fail-closed VRAM watchdog for desktop GPUs.

Project status

Other AMD architectures may work after selecting matching ROCm wheels and changing the build target, but they are not validated by this repository yet. The checked-in build environment currently targets gfx1201 explicitly.

Why this Windows port is different

Upstream vLLM supports AMD ROCm, but its official GPU requirements list Linux and explicitly state that vLLM does not support Windows natively. This fork keeps vLLM's ROCm/HIP execution path and native extensions instead of replacing the engine with llama.cpp, ONNX Runtime, DirectML, or a Vulkan backend.

The comparison below was checked on 2026-08-09. These projects continue to evolve, so follow the links for their latest status.

This project is therefore not the only native-Windows AMD experiment. Its distinctive scope is an integrated, RDNA4-tested, in-tree vLLM ROCm build with operational safety tooling.

How it works

OpenAI client / benchmark
          |
          v
vLLM V1 scheduler, PagedAttention, continuous batching, prefix cache
          |
          +--> Windows process, TCP/ZMQ and event-loop compatibility
          +--> single-rank c10d stand-in (fails if real peers are requested)
          +--> torch.compile / Inductor / Triton
          |
          v
PyTorch ROCm -> HIP / hipBLASLt -> AMD Radeon GPU

Windows GPU performance counters ---> vram_guard.ps1 ---> launched tree only
run-specific log timestamps --------> stall detector -----^

The important porting work includes:

• Building vLLM's C++/HIP extensions with clang-cl, MSVC, CMake, and the ROCm SDK delivered inside AMD's Python wheels.
• Normalizing Windows paths before CMake interprets them and providing MSVC equivalents for POSIX-only source APIs.
• Detecting ROCm devices through torch.cuda when amdsmi is unavailable on Windows.
• Installing a single-rank torch.distributed stand-in because the tested AMD Windows PyTorch wheel has no c10d extension. Operations requiring another rank fail instead of silently returning incorrect results.
• Replacing Unix IPC assumptions with loopback TCP/ZMQ and handling Windows multiprocessing handles correctly.
• Selecting winloop or standard asyncio when POSIX-only uvloop is absent.
• Preferring hipBLASLt on RDNA4. On the test card, the untuned rocBLAS path was dramatically slower for BF16/FP16 GEMMs.
• Keeping Inductor and Triton caches in short paths to avoid Windows MAX_PATH failures.
• Monitoring total dedicated VRAM and log progress independently of vLLM, then terminating only the process tree launched by the watchdog.

See WINDOWS_ROCM_PERFORMANCE.md for measured results, profiles, and current kernel limitations.

Optional Windows multi-GPU transport plugin

This repository deliberately remains a focused Windows port of vLLM. It does not embed the Direct-RCCL/D3D12 transport bridge in the vLLM source tree. The separate, public windows-amd-vllm-multigpu repository installs that bridge as an external plugin.

The plugin must be installed into the same Python environment as this fork and against the exact vLLM commit recorded in the plugin's pin file. Do not use it with stock Linux vLLM or an arbitrary native-Windows vLLM wheel. Its installer fails closed when the source commit, Python environment, ROCm ABI, or bridge binaries do not match. Follow the plugin repository's vLLM host guide; do not copy individual DLLs into this source tree.

The plugin provides one synchronized engine across multiple Windows AMD GPUs; it is not a launcher for unrelated single-GPU servers. Direct-RCCL is the primary process-per-GPU collective path. D3D12 and hybrid routing remain experimental fallbacks with narrower validated message ranges.

Requirements

Hardware and operating system

• Windows 11 x64.
• An AMD GPU supported by the Windows ROCm/PyTorch release you install. Check AMD's current Windows compatibility matrix and HIP SDK system requirements.
• This branch is tested only on gfx1201: Radeon AI PRO R9700 and the same RDNA4 target family.
• Enough system RAM and disk for the checkpoint plus build intermediates. Thirty-two GiB of system RAM is a practical floor for smaller models; 64 GiB or more is recommended for larger checkpoints and source builds.

Software

AMD notes that the complete Linux ROCm stack is not present on Windows. This project uses the Windows-supported HIP/ROCm components packaged with AMD's PyTorch and ROCm SDK wheels. AMD's TheRock releases document the current Windows packages and GPU-specific device-* targets. The tested Torch 2.13/ROCm 10.0 packages are version-pinned. Treat any newer combination as unvalidated until the source build and runtime tests pass.

Do I need Vulkan?

No. This repository uses ROCm/HIP, not Vulkan. Installing the Vulkan SDK will not enable or accelerate vLLM here.

The AMD display driver normally supplies the Vulkan runtime for applications that use it. Install the Vulkan SDK only if you are developing or compiling a separate Vulkan application, such as a Vulkan-enabled llama.cpp build. Do not install Vulkan as a substitute for the ROCm/PyTorch stack above.

Installation

The commands below reproduce the stack tested in this repository. Package indexes change over time; do not mix arbitrary Torch, ROCm, Triton, and vLLM versions and expect ABI compatibility.

1. Install the system tools

Install:

• A compatible AMD Radeon Software driver.
• Python 3.12 x64.
• Git for Windows.
• Visual Studio 2022 Build Tools with Desktop development with C++, MSVC v143, and a Windows 10 or 11 SDK.
• uv from its official installation guide.

Reboot after installing or changing the AMD driver.

2. Clone into a short path

PowerShell:

New-Item -ItemType Directory -Path C:\AI -Force | Out-Null
git clone https://github.com/charlie12345/vLLM_for_AMD.git C:\AI\vllm
Set-Location C:\AI\vllm

A short path matters when Torch/Inductor generates deeply nested cache names.

Automated setup (recommended)

After installing the system tools and cloning the repository, run the repository-local setup from PowerShell:

Set-ExecutionPolicy -Scope Process Bypass
.\setup_windows_rocm.ps1 -PlanOnly
.\setup_windows_rocm.ps1

The default is the validated R9700 target (-GpuArch gfx1201) and eight native build workers. For example, a Radeon RX 7900 XTX is RDNA3 gfx1100, not RDNA4:

.\setup_windows_rocm.ps1 -GpuArch gfx1100 -PlanOnly
.\setup_windows_rocm.ps1 -GpuArch gfx1100

Non-gfx1201 environments use an architecture-suffixed virtual environment so incompatible device libraries and native objects are not silently reused. Accepted targets reflect the ROCm 10 package metadata and vLLM build list; they do not imply model-level validation. See Building for a different AMD architecture.

-PlanOnly checks Windows, the AMD adapter, Git, uv, Visual Studio, system RAM, disk headroom, and repository files without installing packages, opening a GPU context, or building anything. The full run then:

1. Creates or reuses an architecture-isolated Python 3.12 environment.
2. Installs the exact ROCm 10.0, PyTorch 2.13, and Triton packages below.
3. Verifies Torch/HIP and requires the selected GFX target through the fail-closed VRAM guard.
4. Builds and editable-installs the native Windows ROCm extensions.
5. Runs a second guarded vLLM import and device verification.

The script never downloads or launches a model. It does not silently install or update the AMD driver, Visual Studio, Git, or uv; review and install those system-wide prerequisites yourself. The manual commands below remain available for troubleshooting and auditing the automated process.

3. Create the serving environment

uv venv --python 3.12 .venv-rocm10

uv pip install --python .\.venv-rocm10\Scripts\python.exe `
  --extra-index-url https://stable.repo.amd.com/rocm/pytorch/whl-next/ `
  --extra-index-url https://stable.repo.amd.com/rocm/core/whl-next/ `
  --index-strategy unsafe-best-match `
  "torch[device-gfx1201]==2.13.0+rocm10.0.0" `
  "torchvision[device-gfx1201]==0.28.0+rocm10.0.0" `
  "rocm[devel,device-gfx1201]==10.0.0"

uv pip install --python .\.venv-rocm10\Scripts\python.exe `
  "triton-windows==3.7.1.post27" winloop

uv pip install --python .\.venv-rocm10\Scripts\python.exe `
  -r .\requirements\common.txt

uv pip install --python .\.venv-rocm10\Scripts\python.exe `
  "cmake==4.4.2" ninja "packaging>=24.2" `
  "setuptools>=77,<80" "setuptools-scm>=8" `
  "setuptools-rust>=1.9" wheel "jinja2>=3.1.6"

Do not install requirements/rocm.txt wholesale on Windows. It contains Linux-only or unsupported packages such as TileLang, RunAI model streaming, FastSafetensors, AMD Quark, and other optional integrations. The commands above install the tested core serving stack while preserving AMD's ROCm Torch wheel.

4. Verify ROCm before building vLLM

.\.venv-rocm10\Scripts\python.exe -m rocm_sdk test

.\.venv-rocm10\Scripts\python.exe -c `
  "import torch; print(torch.__version__); print(torch.version.hip); print(torch.cuda.get_device_name(0)); print(torch.cuda.get_device_properties(0).gcnArchName)"

For the tested card, the final line must report gfx1201. Stop here if Torch cannot see the expected AMD GPU.

5. Build and install this fork

.\build_windows_rocm.cmd
.\install_windows_rocm.cmd

These scripts load the Visual Studio environment, find ROCm through python -m rocm_sdk path --root, select clang-cl for both C++ and HIP, build for gfx1201, and install vLLM into .venv-rocm10 in editable mode.

Verify the install without loading a model:

New-Item -ItemType Directory -Path .\logs -Force | Out-Null
.\vram_guard.ps1 `
  -LimitGiB 8 -WarnGiB 6 -StallSec 90 `
  -StallLogPath .\logs\verify-runtime.log `
  -Command '".\.venv-rocm10\Scripts\python.exe" ".\tools\verify_windows_rocm_runtime.py" --expected-arch gfx1201 > ".\logs\verify-runtime.log" 2>&1'

Model compatibility

Use a Hugging Face model architecture supported by upstream vLLM, in Safetensors or another native vLLM format. A model being popular does not mean its particular quantization has a fast AMD RDNA4 kernel.

Models tested on this branch

Model-format guidance

As a conservative rule on a display-connected GPU, keep several GiB free for Windows and temporary activations. Checkpoint file size is not the same as peak VRAM use. Model weights, KV cache, activations, compiler workspaces, and the desktop all share the card.

Quick start: guarded OpenAI-compatible server

The launchers default to --gpu-memory-utilization 0.55. The watchdog adds an independent absolute VRAM ceiling.

Open PowerShell in the repository and run:

$root = 'C:\AI\vllm'
$model = 'C:\AI\models\gpt-oss-20b'
$logs = Join-Path $root 'logs'
New-Item -ItemType Directory -Path $logs -Force | Out-Null

$serverLog = Join-Path $logs 'gpt-oss-server.log'
$guardLog = Join-Path $logs 'gpt-oss-server.guard.log'
$command = "`"$root\serve_windows_rocm.cmd`" `"$model`" " +
  "--served-model-name gpt-oss-20b --host 127.0.0.1 --port 8000 " +
  "--max-model-len 2048 --gpu-memory-utilization 0.55 " +
  "-cc.cudagraph_mode=NONE > `"$serverLog`" 2>&1"

& "$root\vram_guard.ps1" `
  -Command $command `
  -LimitGiB 26 `
  -WarnGiB 23 `
  -LogPath $guardLog

For a long-lived server, the example intentionally omits -StallLogPath: an idle API server can legitimately produce no log output. The absolute VRAM limit remains active. Use stall detection for finite benchmarks and conversion jobs.

The guard prints the exact child PID. To stop the server, terminate that exact tree from another terminal rather than killing every Python process on the machine:

$childPid = Read-Host "Enter the child PID printed by vram_guard.ps1"
taskkill /PID $childPid /T /F

Do not close or interrupt the guard and leave its child running. After a forced stop, verify any remaining process by command line before terminating it.

Send a chat request

Run this in a second PowerShell window:

$body = @{
  model = 'gpt-oss-20b'
  messages = @(
    @{ role = 'user'; content = 'Explain PagedAttention in two sentences.' }
  )
  max_tokens = 128
  temperature = 0
} | ConvertTo-Json -Depth 5

Invoke-RestMethod `
  -Uri http://127.0.0.1:8000/v1/chat/completions `
  -Method Post `
  -ContentType 'application/json' `
  -Body $body

Benchmarking

Every model benchmark should use a fresh log and the watchdog:

$stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$log = "C:\AI\vllm\logs\qwen-smoke-$stamp.log"
$guardLog = "C:\AI\vllm\logs\qwen-smoke-$stamp.guard.log"
$command = "C:\AI\vllm\bench_windows_rocm.cmd throughput " +
  "--model C:\AI\models\Qwen3-0.6B --tokenizer C:\AI\models\Qwen3-0.6B " +
  "--dataset-name random --num-prompts 8 " +
  "--random-input-len 256 --random-output-len 64 " +
  "--max-model-len 2048 --gpu-memory-utilization 0.55 " +
  "> `"$log`" 2>&1"

.\vram_guard.ps1 `
  -Command $command `
  -LimitGiB 26 `
  -WarnGiB 23 `
  -StallLogPath $log `
  -StallSec 300 `
  -LogPath $guardLog

The first run for a new model/batch shape may include Triton and Inductor JIT compilation. Repeat the identical workload before calling a result steady state.

VRAM safety

--gpu-memory-utilization and vram_guard.ps1 solve different problems:

• The launcher setting tells vLLM how much GPU memory it may plan to use for weights, activations, graphs, and KV cache. The wrappers inject 0.55 unless the command already specifies a value.
• The watchdog reads Windows' dedicated-GPU-memory performance counter. It sees total adapter usage, including the desktop and other applications, and can terminate only the process tree it launched.

The watchdog fails closed:

• Exit 98: Windows GPU memory counters were unavailable, so the command was not launched.
• Exit 99: the process tree crossed the VRAM limit, stopped producing log output for the configured interval, or the counter repeatedly failed.
• Any other exit code is the launched command's exit code.

Changing the limits for another GPU

You normally do not need to edit vram_guard.ps1. Pass limits per run:

.\vram_guard.ps1 `
  -Command $command `
  -LimitGiB 40 `
  -WarnGiB 36 `
  -StallLogPath $log `
  -StallSec 420 `
  -LogPath $guardLog

Choose values from measurements, not only the advertised card capacity:

1. Measure the idle Windows desktop VRAM use.
2. Reserve at least 4-6 GiB on a display-connected card; reserve more if other GPU applications remain open.
3. Set LimitGiB to the highest total dedicated usage you will tolerate.
4. Set WarnGiB 2-4 GiB below that limit.
5. Start vLLM at --gpu-memory-utilization 0.55, confirm peak usage, then move upward in small steps. More KV cache is not automatically faster.

For example, 26/23 GiB is measured and conservative on the 32 GiB R9700 that also drives the desktop. A headless compute card or a card with more VRAM may use a higher absolute limit, but still needs workspace and OS headroom.

Change the launcher's default without editing it:

$env:WINDOWS_ROCM_GPU_MEMORY_UTILIZATION = '0.65'
$env:WINDOWS_ROCM_MAX_NUM_BATCHED_TOKENS = '4096'
$env:WINDOWS_ROCM_KV_CACHE_DTYPE = 'fp8'
$env:WINDOWS_ROCM_CUDAGRAPH_MODE = 'NONE'

Explicit command-line flags take precedence over these variables.

[!CAUTION] vram_guard.ps1 monitors dedicated GPU memory, not system RAM. Windows may spill GPU allocations into shared system memory before an ordinary CUDA/HIP free-memory query looks obviously wrong. Keep real dedicated-VRAM headroom.

Hang scope and recovery

The watchdog is a best-effort user-mode safeguard. It bounds its own Windows performance-counter and process queries, detects missing log progress, and can terminate only the process tree it launched. It cannot guarantee recovery from a kernel-mode AMD driver, HIP runtime, firmware, PCIe, or desktop compositor hang. No user-space script can make that guarantee.

Save other work and close GPU-heavy applications before a new build or model test. Do not leave an experimental run unattended. If the console still responds, press Ctrl+C once and give the launched processes time to exit. If only the Windows security screen responds, use Ctrl+Alt+Delete and end the launched server/build process; reboot if the desktop or driver does not recover. After a forced restart, confirm both GPUs appear in Device Manager and rerun setup_windows_rocm.ps1 -PlanOnly plus the guarded Torch probe before continuing. Never terminate unrelated Python processes by name.

Useful flags

On a machine with multiple GPUs, select one before starting vLLM:

$env:HIP_VISIBLE_DEVICES = '0'

This selects the one visible GPU used by the base Windows port. It does not enable multi-GPU by itself; install and configure the separate transport plugin described above.

Measured results

Radeon AI PRO R9700, Windows 11, maximum model length 2,048:

These are throughput checks, not model-quality evaluations. See WINDOWS_ROCM_PERFORMANCE.md for methodology and known caveats.

Security and credentials

Do not commit API keys, Hugging Face tokens, model-provider credentials, TLS private keys, .env files, logs, profiles, or downloaded model weights.

This repository's .gitignore excludes common local secret files and generated run artifacts, but ignore rules are not a security boundary. In particular, vram_guard.ps1 records the launched command, so never put a real token in --api-key or another command-line flag.

If the API must require a key, read it interactively into an environment variable before launching the guard:

$secure = Read-Host 'Local vLLM API key' -AsSecureString
$env:VLLM_API_KEY = [System.Net.NetworkCredential]::new('', $secure).Password
Remove-Variable secure

The child process inherits the variable without the value appearing in the guard's logged command. Keep --host 127.0.0.1 unless you have intentionally configured authentication, Windows Firewall, and network access controls.

Before every commit and push:

git status --short
git diff --cached
git grep -n -I -E "(hf_|ghp_|github_pat_|AKIA|BEGIN .*PRIVATE KEY|sk-)" -- .

Review every match; some source identifiers can be false positives. Enable GitHub secret scanning and push protection on the repository as a second layer. If a real credential is ever committed, rotate or revoke it immediately; deleting the working-tree file does not remove it from Git history.

Building for a different AMD architecture

setup_windows_rocm.ps1 -GpuArch <target> selects the matching Torch, torchvision, and ROCm SDK device-<target> extras; exports one architecture to both PyTorch and CMake; and uses a separate environment for nondefault targets. Do not copy an extension wheel built for one GFX target to another.

For manual builds, set both variables before calling the checked-in scripts:

$env:VLLM_ROCM_ARCH = 'gfx1100'
$env:VLLM_VENV = 'C:\AI\vllm\.venv-rocm10-gfx1100'
.\build_windows_rocm.cmd
.\install_windows_rocm.cmd

Find the installed card's target in AMD's support matrix or with a matching working ROCm Torch package. Start with a very small model and conservative guard limits. If changing the architecture in an existing clone, use the architecture-specific environment and a fresh generated build directory; do not reuse native objects from another target.

Architecture support in a ROCm wheel does not guarantee that every vLLM attention or quantization kernel supports that architecture. Treat all targets other than gfx1201 as unverified until they pass model-load, coherent-output, VRAM, and throughput tests.

Known limitations and troubleshooting

• Base port is single-rank: this repository does not bundle a Windows c10d/RCCL transport. Install the separately version-pinned multi-GPU plugin for one synchronized Direct-RCCL/D3D12 engine; do not copy its DLLs into this source tree or use it against an arbitrary vLLM commit.
• Expected amdsmi warning: Windows has no compatible amdsmi package in this stack. The port falls back to Torch for device detection and properties.
• CMake must be new enough: the tested HIP + clang-cl configuration needs CMake 4.4.2. Upstream's ROCm build requirements currently constrain CMake below 4, so do not install that requirements file unchanged.
• Short cache paths matter: the launchers default to C:\AI\vc and C:\AI\vt. Override them with VLLM_CACHE_ROOT and TRITON_CACHE_DIR, but keep them short unless Windows long-path support is enabled.
• FP8-weight graph replay: a tested Qwen3-8B FP8 checkpoint can wedge during HIP graph replay. Use -cc.cudagraph_mode=NONE; the watchdog catches the silent stall.
• Cold-start compilation: new models and batch shapes can trigger Triton or Inductor compilation. Measure the repeated run.
• run-batch frontend: native Windows may require WindowsSelectorEventLoopPolicy for pyzmq, and current cleanup contains a Unix-only SIGKILL call. Output can be valid even if shutdown returns status
  1. The throughput runner exits cleanly.
• Vulkan is not a fallback: if ROCm fails, installing Vulkan does not repair this vLLM backend.
• AWQ compatibility is format-specific: validated native AWQ uses 4-bit weights, group size 32/64/128, FP16/BF16 activations, and no activation-order index. Set VLLM_ROCM_USE_RDNA_W4A16=0 to fall back to generic AutoAWQ while diagnosing an unsupported checkpoint. GPTQ and other INT4 layouts remain kernel-dependent.

Repository scripts

Updating from upstream

This is a focused patch stack on top of vLLM, not an independent inference engine. Upstream changes can alter Python APIs, kernels, dependencies, and build requirements. Check for and apply the latest stable release with:

.\update_windows_rocm.ps1 -CheckOnly
.\update_windows_rocm.ps1

The updater ignores prereleases, fetches the selected tag directly from the official vLLM repository, creates a timestamped backup branch, and rebases the current patch stack. It never pushes or merges. If upstream and Windows changes overlap, it stops at the conflict for review. It also refuses to rebase a non-linear patch stack, because doing so can silently replay already-merged upstream commits as Windows-specific changes.

The scheduled Sync Windows ROCm with upstream vLLM GitHub workflow performs the same check daily. A clean rebase becomes a validation PR; a conflicted rebase becomes an issue. Neither path auto-merges. Review every Windows patch, rebuild the native extensions, and rerun guarded model/quality benchmarks before publishing an update.

Contributors must also follow AGENTS.md and upstream's contribution guide. AI-assisted changes require human review, appropriate tests, and disclosure under the repository's contribution policy.

License and attribution

This repository is based on vllm-project/vllm and retains its Apache License 2.0. See NOTICE for the upstream base and modification attribution.

ROCm, PyTorch, Triton, Hugging Face models, and other downloaded dependencies are separate projects with their own licenses and notices. They are not relicensed by this repository. Review a model's license before downloading, redistributing, or serving it.

If you use vLLM in research, cite the original project:

@inproceedings{kwon2023efficient,
  title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
  author={Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E. and Zhang, Hao and Stoica, Ion},
  booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},
  year={2023}
}

Upstream documentation: https://docs.vllm.ai/