ROCmFP4 for llama.cpp

Experimental AMD-focused FP4 quantization and backend work for llama.cpp, developed on Framework Desktop with AMD Strix Halo 395+ and 128 GB unified RAM.

ROCmFP4 adds new GGUF tensor formats, quantization presets, ROCm/HIP kernels, Vulkan shader support, and reproducible regression guards for long-context MTP inference. The goal is a practical 4-bit format for AMD systems that keeps model coherence protected while improving memory use and decode speed.

Status: experimental research build. Results are hardware-, driver-, model-, and prompt-sensitive. Do not treat these numbers as upstream llama.cpp claims until they are independently reproduced.

What Is ROCmFP4?

ROCmFP4 is a custom 4-bit weight format for GGUF models:

• Q4_0_ROCMFP4: dual-scale 4.50 BPW layout using two finite UE4M3 scale bytes per 32-weight block.
• Q4_0_ROCMFP4_FAST: single-scale 4.25 BPW layout for speed-sensitive tensor roles.
• Tensor-aware presets that mix ROCmFP4 layouts with protected higher-precision tensors where quality matters.
• ROCm/HIP vector-dot, copy, dequant, and FlashAttention handling for the new layouts.
• Vulkan decode and MMQ shader support for the same GGUF tensor types.
• MTP regression guards for long-context target/draft decode.

ROCmFP4 is not MXFP4, NVFP4, or a renamed Q4 format. It uses a Codebook10 4-bit value table and finite unsigned E4M3 half-scale semantics tuned for the current AMD backend paths in this tree.

Proven Local Results

Current strongest reproduced local result:

The benchmark policy is intentionally conservative: a microbenchmark win is not promoted unless end-to-end decode guards hold or improve.

Validation Matrix

Latest Validated Snapshot

• Validated integration snapshot: 4860505ee
• Validation date: 2026-06-13

This snapshot passed the promoted Strix Halo gate with:

• ROCmFP4 quantization tests
• MTMD C API smoke test
• ROCm copy and FlashAttention backend tests
• ROCmFP4 copy, FlashAttention, and runtime regression guards
• Qwen3.6 35B A3B MTP ROCmFP4 smoke test

Documentation

Repository Layout

• ggml/rocmfp4/ — ROCmFP4 format definitions, CPU reference quant/dequant, and HIP helper kernels
• ggml/src/ggml-cuda/ — upstream HIP/CUDA backend files with AMD ROCmFP4 integration (HIP builds use this directory even with -DGGML_CUDA=OFF)
• ggml/src/ggml-vulkan/vulkan-shaders/ — Vulkan shader support for ROCmFP4
• scripts/build-*.sh — build scripts per AMD GPU generation
• scripts/check-rocmfp4-*.sh — correctness and performance regression guards

Build

git clone https://github.com/charlie12345/rocmfp4-llama.git
cd rocmfp4-llama
git checkout mtp-rocmfp4-strix

Pick the script that matches your GPU:

Not sure which GPU you have? See docs/BUILD-AMD-ARCHITECTURES.md for the full gfx target table, runtime environment variables, and Vulkan-only builds.

Strix Halo users: follow docs/STRIX-HALO-QUICKSTART.md for prerequisites, MTP flags, and troubleshooting.

Key binaries after a successful build:

bin/llama-cli
bin/llama-server
bin/llama-quantize
bin/llama-bench
bin/test-backend-ops
bin/test-quantize-fns
bin/test-quantize-perf

Quantize a Model

Start from an F16 or BF16 GGUF source model. Quantizing an already heavily quantized GGUF into ROCmFP4 is useful for smoke tests only; for real quality, use an F16/BF16 source.

Compact Strix profile:

./build-strix-rocmfp4/bin/llama-quantize \
  /path/to/source-bf16.gguf \
  /path/to/model-ROCmFP4-STRIX_LEAN.gguf \
  Q4_0_ROCMFP4_STRIX_LEAN

Quality-biased Strix profile:

./build-strix-rocmfp4/bin/llama-quantize \
  /path/to/source-bf16.gguf \
  /path/to/model-ROCmFP4-STRIX.gguf \
  Q4_0_ROCMFP4_STRIX

Pure experimental formats:

./build-strix-rocmfp4/bin/llama-quantize source.gguf out-dual.gguf Q4_0_ROCMFP4
./build-strix-rocmfp4/bin/llama-quantize source.gguf out-fast.gguf Q4_0_ROCMFP4_FAST

Use the Strix presets for serious testing. The pure FAST format is smaller and can be faster, but may trade away too much coherence on sensitive tensors.

Run a ROCmFP4 Model

Example interactive run:

cd rocmfp4-llama
HSA_OVERRIDE_GFX_VERSION=11.5.1 \
GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
./build-strix-rocmfp4/bin/llama-cli \
  -m /path/to/model-ROCmFP4-STRIX_LEAN.gguf \
  -dev ROCm0 \
  -ngl 999 \
  -c 262144 \
  -b 512 \
  -ub 512 \
  -fa on \
  -ctk q8_0 \
  -ctv q8_0 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-n-min 0 \
  --spec-draft-p-min 0.0 \
  --spec-draft-p-split 0.10 \
  --spec-draft-type-k q4_0 \
  --spec-draft-type-v q4_0 \
  --reasoning on \
  --jinja

For models that do not support MTP or reasoning, remove the --spec-* and --reasoning flags.

Regression Guards

Run the full promoted gate:

env HSA_OVERRIDE_GFX_VERSION=11.5.1 scripts/check-rocmfp4-all-regression.sh

Focused guards:

scripts/check-rocmfp4-quant-regression.sh
scripts/check-rocmfp4-rocm-runtime-regression.sh
scripts/check-rocmfp4-rocm-fattn-regression.sh
scripts/check-rocmfp4-vulkan-runtime-regression.sh
scripts/check-rocmfp4-qwen-mtp-regression.sh
scripts/check-rocmfp4-qwen35-a3b-mtp-regression.sh

The scripts accept environment overrides for model paths, binary paths, context, cache type, MTP settings, and speed floors. See each script for the exact variables.

Design Principles

• Coherence first: tensor-aware presets are preferred over pure speed profiles.
• AMD-specific work must be isolated and measurable.
• ROCm and Vulkan paths must avoid silent fallback where ROCmFP4-specific kernels exist.
• Rejected experiments are recorded so they are not repeatedly rediscovered.
• Claims must cite model, backend, context window, flags, hardware, and date.

Current Limitations

• This is not native FP4 tensor-core execution. rocWMMA FP4 input support is not available in the local headers used by this build.
• ROCmFP4 is currently optimized for AMD Strix Halo behavior and may need retuning on other GPUs.
• TurboQuant and TriAttention are not runtime flags in this isolated tree.
• Some scripts reference local model defaults; override MODEL, ROCMFP4_MODEL, or BASELINE_MODEL for your checkout.

License

This repository is based on llama.cpp and keeps the upstream MIT license. See LICENSE for details. Bundled third-party notices are listed in THIRD_PARTY_NOTICES.md.