llama.cpp (Adaptive KV Streaming & RDNA 4 ROCm Optimizations)

This fork adds Adaptive KV Streaming and RDNA 4 (gfx1201) / ROCm optimizations to llama.cpp. It allows consumer GPUs with 16 GB VRAM (such as the AMD Radeon RX 9070 XT) to execute models like Qwen3.8-27B (IQ4_XS model weights, Q8_0 K / Q4_0 V cache) across their full 262,144-token context window without running out of memory, delivering 80+ t/s decode generation and 1,400+ t/s prefill throughput.

---

Improvements

• Adaptive KV Streaming:
  • Automatically pages and streams the KV cache between host RAM and GPU VRAM.
  • Active VRAM residency is fixed and bounded within dedicated prefill and decode working sets.
  • Dedicated asynchronous copy engine streams KV pages via PCIe SDMA alongside compute streams.
  • Enables full 262,144-token context depth on a single 16 GB GPU with 100% recall and zero allocation crashes.
• RDNA 4 (gfx1201) and ROCm Kernel Optimizations:
  • Multi-column mmvq with 4 rows per block tuned for RDNA 4 wave dispatch.
  • Kernel fusions: fused residual add + RMS norm, fused SwiGLU + quantized activation copies for mmvq, and fused gated delta net (GDN) recurrent kernels.
  • Sibling matmul activation reuse across parallel projections.
  • Aligned LDS reads for K fragments and compact V scales in paged decode attention.
  • Vectorized 16-byte reads per call for Q4_K and Q5_K mmvq.
• Probabilistic Rejection-Sampled Speculative Pipeline:
  • Multi-Token Prediction (MTP) drafts are sampled probabilistically at request temperature and verified by rejection sampling instead of naive greedy matching, preserving model output distribution.
  • Single-head MTP decodes verify catch-up in its first draft step.
  • Stacks opportunistic n-gram drafting with MTP, delivering up to 16-token speculative bursts on repetitive context and code editing.
• HIP Runtime and Graph Tuning:
  • Tuned for ROCm 7.2.3 and HIP graphs with single-queue async mode to prevent launch bubbles.
  • Constrains hardware queue allocations to prevent preemption save area VRAM blowups during long prefills.

This work builds on concepts from RaymondHuang210129/llama.cpp-adaptive-kv-streaming and stew675/llama-cpp-rdna-boosts, re-implemented specifically for RDNA 4 hardware targets in ROCm/HIP.

---

Setup

The configurations and benchmarks in this repository use Qwen3.8-27B (ASCII-condensed IQ4_XS, 12.63 GiB): bsaleh03/Qwen3.8-27B-ASCII-Condensed. Another option would be troed/Qwen3.8-27B-ASCII-Condensed.

System Requirements

• GPU: AMD Radeon RX 9070 XT (gfx1201, RDNA 4) or compatible ROCm GPU with >= 16 GB VRAM.
• Host Bus: PCIe 5.0 x16 recommended for optimal streaming throughput at > 80k context (PCIe 4.0 supported).
• Host RAM: >= 32 GB RAM (~7.4 GB host memory allocated for 262,144 context; dual-channel DDR5 recommended).
• ROCm Toolchain: ROCm 7.2.3 or later on Linux.

NixOS

nix run .#serve-openai -- -m /path/to/Qwen3.8-27B-ASCII-Condensed-UD-IQ4_XS.gguf

Other Linux Distributions

Follow official documentation to install AMD ROCm for your distribution.

# Build
cmake -B build-rocm -G Ninja \
  -DCMAKE_C_COMPILER=hipcc \
  -DCMAKE_CXX_COMPILER=hipcc \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS=gfx1201 \
  -DGGML_HIP_NO_VMM=ON \
  -DGGML_CUDA_FA_QUANTS="q4_0-q4_0,q8_0-q8_0,f16-f16,bf16-bf16,q8_0-q4_0" \
  -DGGML_BACKEND_DL=OFF

cmake --build build-rocm --config Release -j$(nproc)

# Launch server
export ROCR_VISIBLE_DEVICES=0
export GPU_MAX_HW_QUEUES=2
export DEBUG_HIP_FORCE_ASYNC_QUEUE=1
export DEBUG_HIP_FORCE_GRAPH_QUEUES=1

./build-rocm/bin/llama-server \
  -m /path/to/Qwen3.8-27B-ASCII-Condensed-UD-IQ4_XS.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 262144 -np 1 -dev ROCm0 -fa on -fit off \
  -ctk q8_0 -ctv q4_0 -ctkd q8_0 -ctvd q4_0 \
  --kv-stream \
  --kv-resident-mib 450 \
  --kv-resident-mib-decode 2100 \
  --spec-type draft-mtp,ngram-mod \
  --spec-ngram-mod-n-max 16 \
  --spec-ngram-mod-n-min 12 \
  --spec-ngram-mod-n-match 24 \
  --spec-draft-sampling probabilistic \
  -b 2048 -ub 512

---

Benchmark

• Hardware: AMD Radeon RX 9070 XT (gfx1201), PCIe 5.0 x16, ROCm 7.2.3.
• Fork Build: Commit bd063a49a (feature/adaptive-kv-stream-rocm).
• Upstream Build: Commit 23f7aabc1 (upstream/master), compiled with identical ROCm flags.
• Workload: Real SQLite amalgamation C source code (sqlite3.c) scaled to context ladder depths (1k, 4k, 16k, 64k, 128k, 256k), prompting a standard helper function implementation (sqlite3_custom_column_hash).
• Generation: 128 output tokens at greedy temperature 0.0 (deterministic token path).
• Metrics: Prompt processing throughput (prefill t/s) and token generation throughput (decode t/s) reported directly by the server /completion JSON timings.

Decode Throughput (tokens/second)

Prefill Throughput (tokens/second)

---

Multi-Turn Agent Performance

I have been using this setup for multi-turn agentic coding sessions. Here are the aggregated results from these sessions:

While the measurements above were taken at 300W TDP, operating with a 231W power limit (-24% power reduction relative to 304W default TDP) incurs only a 3.6% prefill latency and exhibits virtually identical decode throughput, providing significant power and thermal savings for real-world workloads.

---

(Original upstream llama.cpp documentation follows below)

---

llama.cpp

Quick start

A few options to get llama.cpp installed on your machine:

# curl
curl -LsSf https://llama.app/install.sh | sh

# powershell
irm https://llama.app/install.ps1 | iex

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models
• Release process

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
• sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain