LLM inference in C/C++
See the code
llama.cpp for AMD Strix Halo
upstream llama.cpp / ggml / halo-box/llama.cpp
The goal is simple: more functionality, and the fastest llama.cpp for strix around. And help the community with a single fast llama.cpp fork instead of many competing ones.
A community fork of llama.cpp for AMD Strix Halo - the Ryzen AI Max / Max+ 300 series
APUs (gfx1151, RDNA 3.5 integrated GPU, up to 128 GB of unified LPDDR5X shared between CPU and GPU).
Strix Halo is an unusual target. It has more addressable memory than almost any consumer discrete GPU, and far less bandwidth; the iGPU shares its memory controller with the CPU; and both the Vulkan (RADV) and ROCm/HIP paths have RDNA 3.5 specific behaviour that upstream has no hardware to reproduce. Changes that only make sense on this one device - or that need a lot of measurement on it before they are ready to propose anywhere else - live here.
Practically, that means this repo has its own rules - most visibly, AI coding agents may open pull requests here. See CONTRIBUTING.md and AGENTS.md.
Build from source. Two backends are worth using on Strix Halo:
Vulkan (RADV on Mesa; the easiest path, and the best one for most models)
cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
ROCm / HIP (needs ROCm installed; build for gfx1151 explicitly)
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
Then:
# chat, pulling the model straight from Hugging Face
./build/bin/llama-cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# OpenAI-compatible API server + web UI on http://localhost:8080
./build/bin/llama-server -hf ggml-org/Qwen3.5-0.8B-GGUF
Full build documentation, including Windows and Docker, is in docs/build.md.
Give the iGPU enough memory. The APU's memory is shared, and the GPU can only use what the firmware and kernel let
it map. Two things control this: the UMA / dedicated-VRAM split in your BIOS, and the amdgpu GTT limit on Linux
(amdgpu.gttsize, in MB, and ttm.pages_limit, in 4 KB pages, as kernel command-line parameters). Which of those you
need depends on your kernel version - newer kernels size GTT more generously on their own. If a model that clearly
fits in RAM fails to allocate, this is almost always why.
ROCm and batched inference. On gfx1151 there is an async-execution correctness bug in the HIP path: batched
inference can return badly wrong output (perplexity ~88 against ~9.4 for the same model). Setting
HIP_LAUNCH_BLOCKING=1 serializes kernel launches and restores correctness, at a performance cost. Our ROCm CI runs
with it set. It is a workaround for a ROCm/HIP issue, not a fix, and it should go away when that is fixed upstream.
Vulkan mat-vec chunking. Batched mat-vec at 3, 5 and 6 columns is several times slower than at 1, 2 and 4 on RADV
here, which hits speculative decoding hard (it verifies at exactly those batch sizes). This fork splits such batches
into column counts that are measured to scale. Set GGML_VK_MMV_NO_SPLIT=1 to restore the single upstream dispatch,
e.g. to compare against it.
Measure things. GGML_VK_PERF_LOGGER=1 (any value) gives per-op timings on the Vulkan backend and is how most of the findings
above were made. llama-bench and llama-perplexity are the tools for before/after numbers, and PRs here are
expected to carry them - see Benchmarking requirements.
Everything else is upstream llama.cpp. The additions currently carried here:
Strix Halo only (lives here, measured on gfx1151)
| Change | Flag / switch | What it does |
|---|---|---|
| Speculative prefill | --spec-prefill | A small draft model scores prompt tokens by attention importance so the target model only prefills the ones that matter, cutting time-to-first-token on long prompts |
| Adaptive speculative draft length | --spec-draft-adaptive | Sizes each draft from a measured per-sequence acceptance EMA rather than always drafting --spec-draft-n-max; speeds up MTP and DFlash |
| Vulkan fixes and tuning for RDNA 3.5 | Driver-gated coopmat LDS stride padding, UMA bulk readback gated on host-cached mappings, IQ3_S mat-vec at batch sizes > 4, and a radix top-k kernel for large k | |
| Hidden server presets | hidden in the models .ini | Keep a model loadable by name while omitting it from GET /models |
| Multi-point reasoning budget | --reasoning-budget-* (upstream has the hard budget only) | Intro message, two soft warnings, a grace period to finish a paragraph after the budget runs out, and reasoning-token usage telemetry |
| Vulkan batched mat-vec chunking | GGML_VK_MMV_NO_SPLIT=1 to disable | Dispatches batched mat-vec at the column counts that actually scale on RDNA 3.5, instead of the slow NUM_COLS shader variants |
| ROCm/HIP quantized matmul on RDNA 3.5 | MMQ/MMVQ tile configurations and register prefetching, compact MUL_MAT_ID with quant- and shape-specific tiles for 256-expert MoE prefill, fused activation quantization for Q8_0 and Q6_K decode | |
| ROCm/HIP MoE decode fusion | GGML_CUDA_DISABLE_WEIGHTED_DOWN=1, GGML_CUDA_DISABLE_MMID_512=1 | Fused routing, weighted expert reduction and shared-expert gate for Qwen3.5/3.6 and Ling style MoE |
| ROCm/HIP Gated DeltaNet | GGML_CUDA_DISABLE_GDN_GATE=1 | DPP reductions and a tiled multi-column kernel for prefill; the whole conv -> norm -> gate -> recurrence -> state copy decode chain as one kernel |
| ROCm/HIP grouped decode matvecs | GGML_CUDA_DISABLE_MMV_GROUP=1 | Consecutive single-column matvecs that read the same activation (gate/up pairs, hyper-connection projections) launch as one kernel |
| ROCm/HIP flash attention on RDNA 3.5 | WMMA path for D=256 prefill at depth, Q8_0 KV tile kernel for decode | |
| Speculative checkpoints on device | llama-server keeps speculative-decoding checkpoints in device memory instead of copying them to the host | |
| ROCmFPx quant types | llama-quantize types Q4_0_ROCMFP4, Q4_0_ROCMFP4_FAST, Q2/Q3/Q6/Q8_0_ROCMFPX and the _LEAN/_COHERENT/_STRIX recipes | Loads the ROCmFP4 GGUFs published for Strix Halo. CPU codecs plus Vulkan dequant, mat-vec, matmul and integer-dot kernels. Weight formats only: not accepted as KV-cache types |
| Repeatable output at depth | Freed KV cells are zeroed so masked-out rows never carry stale K/V, and the Vulkan radix top-k assigns output slots deterministically |
Every ROCm/HIP change above is guarded on architecture, shape and layout, so other devices see upstream behaviour.
Run --help, or see tools/server/README.md, for the full options.
The ones that matter on this hardware:
| Backend | Notes |
|---|---|
| Vulkan | RADV on the RDNA 3.5 iGPU - the default recommendation |
| HIP | ROCm on gfx1151 - see the HIP_LAUNCH_BLOCKING note above |
| CPU | Zen 5 cores with AVX-512 - useful for offloading part of a model, though it shares bandwidth with the iGPU |
| ZenDNN | AMD CPU acceleration |
| RPC | Distribute a model across several machines |
llama.cpp supports many more (CUDA, Metal, SYCL, CANN, OpenCL, WebGPU, ...); they are all still present in the tree
and unmodified. See the upstream README for that list.
This is a small community project. Strix Halo owners with a benchmark, a bug report, or a patch are exactly who it is for.
CI runs the standard llama.cpp suite plus a self-hosted gfx1151 ROCm job on real hardware.
This project is a fork and owes everything to the people who built what it forks:
llama-server - MIT licenseLLM inference in C/C++
See the code
llama.cpp for AMD Strix Halo
upstream llama.cpp / ggml / halo-box/llama.cpp
The goal is simple: more functionality, and the fastest llama.cpp for strix around. And help the community with a single fast llama.cpp fork instead of many competing ones.
A community fork of llama.cpp for AMD Strix Halo - the Ryzen AI Max / Max+ 300 series
APUs (gfx1151, RDNA 3.5 integrated GPU, up to 128 GB of unified LPDDR5X shared between CPU and GPU).
Strix Halo is an unusual target. It has more addressable memory than almost any consumer discrete GPU, and far less bandwidth; the iGPU shares its memory controller with the CPU; and both the Vulkan (RADV) and ROCm/HIP paths have RDNA 3.5 specific behaviour that upstream has no hardware to reproduce. Changes that only make sense on this one device - or that need a lot of measurement on it before they are ready to propose anywhere else - live here.
Practically, that means this repo has its own rules - most visibly, AI coding agents may open pull requests here. See CONTRIBUTING.md and AGENTS.md.
Build from source. Two backends are worth using on Strix Halo:
Vulkan (RADV on Mesa; the easiest path, and the best one for most models)
cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
ROCm / HIP (needs ROCm installed; build for gfx1151 explicitly)
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
Then:
# chat, pulling the model straight from Hugging Face
./build/bin/llama-cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# OpenAI-compatible API server + web UI on http://localhost:8080
./build/bin/llama-server -hf ggml-org/Qwen3.5-0.8B-GGUF
Full build documentation, including Windows and Docker, is in docs/build.md.
Give the iGPU enough memory. The APU's memory is shared, and the GPU can only use what the firmware and kernel let
it map. Two things control this: the UMA / dedicated-VRAM split in your BIOS, and the amdgpu GTT limit on Linux
(amdgpu.gttsize, in MB, and ttm.pages_limit, in 4 KB pages, as kernel command-line parameters). Which of those you
need depends on your kernel version - newer kernels size GTT more generously on their own. If a model that clearly
fits in RAM fails to allocate, this is almost always why.
ROCm and batched inference. On gfx1151 there is an async-execution correctness bug in the HIP path: batched
inference can return badly wrong output (perplexity ~88 against ~9.4 for the same model). Setting
HIP_LAUNCH_BLOCKING=1 serializes kernel launches and restores correctness, at a performance cost. Our ROCm CI runs
with it set. It is a workaround for a ROCm/HIP issue, not a fix, and it should go away when that is fixed upstream.
Vulkan mat-vec chunking. Batched mat-vec at 3, 5 and 6 columns is several times slower than at 1, 2 and 4 on RADV
here, which hits speculative decoding hard (it verifies at exactly those batch sizes). This fork splits such batches
into column counts that are measured to scale. Set GGML_VK_MMV_NO_SPLIT=1 to restore the single upstream dispatch,
e.g. to compare against it.
Measure things. GGML_VK_PERF_LOGGER=1 (any value) gives per-op timings on the Vulkan backend and is how most of the findings
above were made. llama-bench and llama-perplexity are the tools for before/after numbers, and PRs here are
expected to carry them - see Benchmarking requirements.
Everything else is upstream llama.cpp. The additions currently carried here:
Strix Halo only (lives here, measured on gfx1151)
| Change | Flag / switch | What it does |
|---|---|---|
| Speculative prefill | --spec-prefill | A small draft model scores prompt tokens by attention importance so the target model only prefills the ones that matter, cutting time-to-first-token on long prompts |
| Adaptive speculative draft length | --spec-draft-adaptive | Sizes each draft from a measured per-sequence acceptance EMA rather than always drafting --spec-draft-n-max; speeds up MTP and DFlash |
| Vulkan fixes and tuning for RDNA 3.5 | Driver-gated coopmat LDS stride padding, UMA bulk readback gated on host-cached mappings, IQ3_S mat-vec at batch sizes > 4, and a radix top-k kernel for large k | |
| Hidden server presets | hidden in the models .ini | Keep a model loadable by name while omitting it from GET /models |
| Multi-point reasoning budget | --reasoning-budget-* (upstream has the hard budget only) | Intro message, two soft warnings, a grace period to finish a paragraph after the budget runs out, and reasoning-token usage telemetry |
| Vulkan batched mat-vec chunking | GGML_VK_MMV_NO_SPLIT=1 to disable | Dispatches batched mat-vec at the column counts that actually scale on RDNA 3.5, instead of the slow NUM_COLS shader variants |
| ROCm/HIP quantized matmul on RDNA 3.5 | MMQ/MMVQ tile configurations and register prefetching, compact MUL_MAT_ID with quant- and shape-specific tiles for 256-expert MoE prefill, fused activation quantization for Q8_0 and Q6_K decode | |
| ROCm/HIP MoE decode fusion | GGML_CUDA_DISABLE_WEIGHTED_DOWN=1, GGML_CUDA_DISABLE_MMID_512=1 | Fused routing, weighted expert reduction and shared-expert gate for Qwen3.5/3.6 and Ling style MoE |
| ROCm/HIP Gated DeltaNet | GGML_CUDA_DISABLE_GDN_GATE=1 | DPP reductions and a tiled multi-column kernel for prefill; the whole conv -> norm -> gate -> recurrence -> state copy decode chain as one kernel |
| ROCm/HIP grouped decode matvecs | GGML_CUDA_DISABLE_MMV_GROUP=1 | Consecutive single-column matvecs that read the same activation (gate/up pairs, hyper-connection projections) launch as one kernel |
| ROCm/HIP flash attention on RDNA 3.5 | WMMA path for D=256 prefill at depth, Q8_0 KV tile kernel for decode | |
| Speculative checkpoints on device | llama-server keeps speculative-decoding checkpoints in device memory instead of copying them to the host | |
| ROCmFPx quant types | llama-quantize types Q4_0_ROCMFP4, Q4_0_ROCMFP4_FAST, Q2/Q3/Q6/Q8_0_ROCMFPX and the _LEAN/_COHERENT/_STRIX recipes | Loads the ROCmFP4 GGUFs published for Strix Halo. CPU codecs plus Vulkan dequant, mat-vec, matmul and integer-dot kernels. Weight formats only: not accepted as KV-cache types |
| Repeatable output at depth | Freed KV cells are zeroed so masked-out rows never carry stale K/V, and the Vulkan radix top-k assigns output slots deterministically |
Every ROCm/HIP change above is guarded on architecture, shape and layout, so other devices see upstream behaviour.
Run --help, or see tools/server/README.md, for the full options.
The ones that matter on this hardware:
| Backend | Notes |
|---|---|
| Vulkan | RADV on the RDNA 3.5 iGPU - the default recommendation |
| HIP | ROCm on gfx1151 - see the HIP_LAUNCH_BLOCKING note above |
| CPU | Zen 5 cores with AVX-512 - useful for offloading part of a model, though it shares bandwidth with the iGPU |
| ZenDNN | AMD CPU acceleration |
| RPC | Distribute a model across several machines |
llama.cpp supports many more (CUDA, Metal, SYCL, CANN, OpenCL, WebGPU, ...); they are all still present in the tree
and unmodified. See the upstream README for that list.
This is a small community project. Strix Halo owners with a benchmark, a bug report, or a patch are exactly who it is for.
CI runs the standard llama.cpp suite plus a self-hosted gfx1151 ROCm job on real hardware.
This project is a fork and owes everything to the people who built what it forks:
llama-server - MIT license