gfx906 fork (this repository)

TLDR: +37% prefill, +16% 120k-deep fill (direct same-config A/B), 250k context for a 27B Q6_K on 40 GB VRAM, DFlash2 speculative decoding (+43% TG vs what mainline can do here), near-identical outputs vs upstream master - for gfx906 (Radeon VII / MI50) and mixed ROCm + Vulkan rigs. Last upstream sync: 2026-09-24 (merge 7887b8887, lane-gated; master carries it all). Agent workloads can opt into exact prompt-message deduplication to avoid re-prefilling repeated large tool, user, or system messages.

What this is: a production llama.cpp fork for AMD gfx906 GPUs (Radeon VII / MI50 / MI60) and mixed ROCm + Vulkan rigs. Upstream has no gfx906 tuning and no controllable pipeline parallelism for heterogeneous tight-VRAM fits. Reference deployment: Qwen3.8-27B (i1-Q6_K) + DFlash2 speculative decoding at 250k context on 40 GB VRAM (2x Radeon VII + GTX 3080), deterministic lane outputs.

How much it helps (vs upstream master 2026-09-09, same-config A/B)

Direct naked pair (identical common-subset args, -c 130000 -sm layer -ts 35,20,45, f16-K/f16-V, NO speculative decoding on either side; bench/ab-mainline-0909.sh, E140): isolates pure fork kernels/tunes. Prod pair = fork's full DFlash2 deployment vs mainline naked (mainline has no DFlash; also cannot fit 250k on this rig at all).

metric	mainline t/s	fork t/s	gain
prefill PP16384 (naked pair)	320.0	439.6	+37%
120k deep fill (naked pair)	241.1	280.7	+16%
TG @120k depth (naked pair)	9.8	9.5	parity (fork's TG edge is spec-decode)
TG @120k (prod configs)	9.8 (no spec available)	14.0 (DFlash2 n4)	+43%
context	cannot fit 250k	250k on 40 GB	tight-fit machinery
outputs	-	-	KLD ~5e-4 mean, same-top 99.14% (E140)

Output identity and quality vs upstream

2026-09-09 refresh (E140, both builds on the same upstream base 434ddbbc0: fork build-sync0909 vs mainline ~/dev/llama.cpp build-0909; wikitext-2 10k ctx, bench/ab-mainline-0909.sh):

metric	mainline	fork	verdict
perplexity (29 chunks, c=10000)	6.2785 +/- 0.04076	6.2777 +/- 0.04074	tie (fork epsilon better, in-noise)
KLD mean / same-top (cross, 6 chunks)	-	~0.0005 / 99.140%	50x tighter than the 2026-09-06 pair
temp-1.0 lane reproducibility	repro_ok	repro_ok	both deterministic

Historical 2026-09-06 pair (E122-E124, fork build-dflash-novega vs mainline-cmp c5a5535e6, BEFORE the shared 2026-09-09 numerics base; full PPL 65-chunk lane + logit KLD tables in journal/JOURNAL-2026-09-06.md):

metric	upstream	fork	verdict
perplexity (65 chunks)	1.3033 +/- 0.0056	1.3043 +/- 0.0056	identical within noise
PPL pass speed (c=4096)	19.6 s	14.7 s	fork +25% (note: at c=10000 on 2026-09-09 builds mainline passes are 13% faster - batch-shape specific, under investigation)
KL divergence, mean / median	0	0.0248 / 1.6e-6	tail-driven
KL divergence, p95 / p99 / max	0	0.010 / 0.31 / ~16	near-tie tail only
top-1 agreement	100%	98.65%	flips on near-ties only
upstream top-1 in fork top-5	100%	99.81%	sampling-relevant
logit RMSE, full vocab / top-100	0	0.067 / 0.50	quant-floor + rare flips

Attribution (E123, archive-build probe series): build flags and toolchain exonerated (exact zero); divergence is the vega MMQ tune set's accumulation order (E82 restoration, signature stable and unchanged since 2026-09-02); upstream sync, q6k64 tile, draft mirror, GDN and q8_1 ports are numerics-neutral. Sampled outputs (temp 1.0 prod config) unaffected; rare token flips possible only on long temp-0 runs over flat distributions.

Who benefits

You run...	You get...
gfx906 / MI50 / Radeon VII	wave64-correct kernel tables, sha-gated
mixed ROCm + Vulkan	controllable pipeline parallelism + safe fallback
heterogeneous GPU setups	unlike GPUs in one pipeline: weighted layer splits, cross-backend scheduling, per-device fits
Qwen3.8 / qwen4exp	DFlash2 drafter, adaptive depth, GDN prefill
coding agents with repeated tool output	opt-in exact message dedup that preserves the first copy and prompt-cache prefix
anyone reproducibility-minded	every change gated on unchanged temp-0 sha

What we changed, and why

• wave64 kernel tables (fork + upstream #27841's GCN row): MMQ tiles (Q6_K I=64), top-k, fattn dispatch. Occupancy 1 -> 2 waves hides latency -> +20% PP.
• Draft head mirror (LLAMA_DFLASH_MIRROR_OUTPUT=1 + --spec-draft-device): device-local copy of the borrowed vocab head, single-device draft graph -> draft rounds -42%, TG +11% @120k depth.
• Pipeline parallelism: fork-only on/off switch, safe fallback, draft-ctx exclusion (controls + per-device fit refinements from the eaman patch store, store.piffa.net/lm/bug; A/B-adopted 2026-08-24). Benefit: no startup aborts or capture corruption on tight fits, honest fit tables.
• Tight fits: f16-K/q8_0-V KV, fattn path selector, frugal buffers (compute 4.7x smaller than stock), HIP graph robustness. Benefit: the 250k-on-40GB fit itself - stock cannot load it.
• Robustness ports, bit-exact (mx-llama.cpp survey): fattn-vec stride fix, pipeline drain before seq-layout reset, HIP graph exec reinstantiation. Benefit: stability on tight-fit + HIP-graph regimes. (GDN chunked prefill and q8_1 activation cache also carried, env-gated; measured perf-neutral here.)
• Checkpoint rewind (#26004 + upstream #28302, adopted 2026-09-09): slot save/restore with context-checkpoint persistence and eviction fixes - prompt rewind after save/restore drops ~455 s re-prefill to <1 s (T2 probe: 23 tokens reprocessed vs 868, bit-exact output).
• MoE MMQ OOB fix (host-side padding, 2026-09-09): fixes out-of-bounds reads in expert-routed MMQ at flattened sizing; PPL tie, perf-neutral.
• Prompt message deduplication (llopresto87 commit 02957b773, adopted 2026-09-29): llama-server can replace later byte-identical large messages with references to their first copy. It is off by default; enable it with --message-dedup and tune it with --message-dedup-min-bytes and --message-dedup-roles. See README_DEDUPLICATION.md.
• gfx906 packed-byte subtraction (upstream #29478, adopted 2026-09-29): __vsubss4 -> __vsub4 in fattn-vec V-dequant, MMQ tile loads and q3_K/q6_K MMVQ. Operands are bounded so saturation never triggers; TG +9-10% on the Q6_K DFlash lane with identical temp-0 SHA. Evidence: bench/logs/upstream-prs-20260929/summary.md.
• Upstream syncs: full merges, each lane-gated on sha + perf. Benefit: current fixes ride along at zero measured cost.

What did not work (measured)

idea	why out
MMQ inner-loop restructures, 5 variants (fork probes)	~41% issue ceiling is structural on gfx906
fattn scheduling/geometry probes (fork lanes)	compiler schedule is a local optimum (-0.5 to -5.8%)
deeper draft n_max=5 (fork lane)	6-row verify hits a kernel-shape cliff (-28% TG)
K quantization beyond q8_0-V (fork measurement, E54)	native tile -2.6 t/s, depth effect only ~5%
native q8_0-K/q4_0-V fattn tile as default (E184-E190)	bit-canonical after geometry mirror, but slower than convert at every depth (16k -3%, 160k fill -15%, 160k TG -9%); stays a certified fallback
ts rebalance / drafter relocation (fork lanes, E105 + 08-15 sweeps)	decode is overhead-bound, not bandwidth-bound
checkpoint sparsify / off (fork lanes, AB6 + E119.4)	neutral / breaks prompt restore
WY-chunked GDN prefill, software tiles (upstream #26001 port, E137)	x0.5 vs recurrent - gfx906 has no matrix units; op is latency-bound at 2.6% FP32 peak
GDN chain-shortening family (micro-chunk CS=5, pair-pipelining, E138)	GDN wall shares too small: 7.6% of fill, 2% of TG round - ceilings under campaign threshold
full survey (adopted/rejected per PR)	table below + journal/ + bench/FINDINGS.md

Upstream PRs and projects evaluated

name	what was tested	why not adopted
upstream #27173 (PatrickWalther) draft-chain perf	full triage + our graphs-off lane	launch overhead TG-neutral here; dflash already one-decode; output-head mirror taken instead
upstream #21698 (iacopPBK) q8_0 loader remap	ported + A/B, Q8LDR build	+1.0% = noise; PR gains measured under upstream's RDNA2-fallback tiles, not our vega table
upstream #23685 (ravel7524) packed Q8_1 MMVQ	trial merge + confound screen	no gain on the target path; parked as reference for the Q4_K drafter
upstream #24546 (ravel7524) MoE expert-width N-tiles	triage vs PMU attribution	96.9% of MMQ cycles already run J=64 fb=0 - no headroom
upstream #28313 (pwilkin) ROCm top-k resolve	triage vs E53 gate	top-k worth <= 0.4 t/s in TG
upstream #28178 (GiuseppeCapaldo93) small-D2D copy kernel	live-graph copy probe	no per-layer-per-token small copies on our path
upstream #25635 (ynankani) fattn XOR swizzle	fork probes of the same schedule	-0.5 to -5.8%; compiler schedule already a local optimum
upstream #28102 (pwilkin) fattn tuning (gfx1201)	geometry probes	wrong target arch; same local-optimum result
upstream #27694 (praneshgo) probabilistic drafter	triage	rejection-sampling verify = sha-breaking; no dflash support
upstream #28305 (ynankani) static sampling subgraph	triage	dflash2 walks the lattice on CPU; no backend sampling
upstream #28390 (am17an) drafter meta-backend	triage	VRAM saving only under -sm tensor; we run -sm layer
upstream #28391 (ggerganov) default spec config	triage	perf-neutral; re-verify additive flag semantics at next sync
upstream #28333 (mxxm-t) zero MTP carrier	triage	MTP-family fix; dflash carrier is separate - check at sync
upstream #27692 (kilofox) speculative prefill	triage	lossy prefill; watch
upstream #21170 (uaruss) ROCm multi-GPU IMA fix	hand-applied for crash tests	fixes recurrent-state restore; our crash class was device-context D2D
upstream #21849 (avialallon) per-arch MMQ tiles	triage	converges with our vega table; drop our gate only after it merges
upstream #26592 (Geramy) hipCUB top-k	triage	no hot sort on our path
mx-llama.cpp (mxxm-t)	full survey	3 robustness ports adopted (bit-exact); GDN chunk + q8_1 cache perf-neutral; TP/AllReduce, layout cache, replay coupling not our topology
eaman patch store (store.piffa.net/lm/bug)	full survey	pipeline-parallel controls + fit refinements adopted 2026-08-24; rs sync line wedges on abort
llopresto87/llama.cpp-prompt-dedup	ported commit 02957b773 with its conformance and scenario suites	adopted 2026-09-29; opt-in and stateless, with no prompt change when disabled

| upstream #29191 (streaming load-mode) | evaluated | UMA-only design + 24G swap requirement; not applicable to dGPU rigs | | upstream #28414 (prefetch-experts-slots) | evaluated vs our staging ring | ring already frees the host timeline; in-order queue caps device overlap (transfer-queue probe: 222 vs 248.4 t/s) | | upstream #27851 (tiled k-quant CPU mulmat) | cherry-picked, GGML_CPU_TILED_MM=0 | not bit-exact; Q6_K GEMV regression; true-CPU mode still loses to vk streaming (74 vs 248 t/s, no VNNI here) | | upstream #28118 (spec ckpts on-device) | ported + llama_state_seq_evict fix, GGML_SPEC_CKPT_ON_DEVICE 0/draft/1 | steady TG 10.2 -> ~12 in full mode; eviction at big-prompt start keeps PP-after-TG clean (257 vs 199); promoted to fast lane | | upstream #29274 (NVIDIA 4-row rm_id) | ported, GGML_VK_NV_RMID_ROWS | TG within noise-to-positive, PP clean; opt-in | | upstream #29353 (GDN chunked kernel, hip) | triage | GDN share too small on the flash lane; gfx906 matrix-unit limits; watch |

Evidence and build

Every claim has a lane row: journal/ (experiment log), bench/ (A/B harness, FINDINGS.md, vram-test, rocprof stack). Build: ./build-dflash-novega.sh (Vulkan + ROCm 6.1 - pinned, the measured optimum for gfx906 here; launchers also pin its userspace). Current prod build: build-dedup/bin (0.5.0-dev, full prior fork state plus opt-in prompt message deduplication, promoted 2026-09-29 after matched PP16384/TG1024/temp-0 SHA gates); production launchers point there by default (rollback: build-rcfix/bin).

llama.cpp

Quick start

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models
• Release process

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
• sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain