llama.cpp (ROCmFP4 port)

[!NOTE] What this repository is: a port of the official ggml-org/llama.cpp that adds the ROCmFP4 (STRIX) GGUF quantization format, built on top of the official DFlash2 speculative-decoding branch (PR #27342 by SubSir). Everything else tracks upstream (MIT).

Fork additions (details, rationale and measured numbers in docs/rocmfp4.md):

• ROCmFP4 (STRIX) GGUF quantization — Q4_0_ROCMFP4 / Q4_0_ROCMFP4_FAST (type 100/101): 32-value blocks, split-nibble layout (low nibbles = values 0..15, high nibbles = values 16..31), Codebook10 values, half-scale UE4M3 scales. Full CPU + HIP kernels: MMVQ, MMQ (incl. RDNA3 WMMA path), dequantization, MUL_MAT_ID. Without this port, official llama.cpp cannot load ROCmFP4 models at all.
• Reference serving setup for RX 7900 XTX — scripts/serve-dflash2.sh: fp4 weights + q4_0 KV cache (LLAMA_ATTN_ROT_DISABLE=1), DFlash2 draft (n5/p0.4) stacked with ngram-map-k4v (n12/m48), 256K context at 24 GiB (--no-kv-unified --ctx-checkpoints 2 --cache-ram 4096). Measured on Qwen3.8-27B: 41 tok/s fp4 baseline → 68 (reasoning content) → 98 tok/s (repetitive content, ngram hits). Speculative acceptance is content-driven — creative prose stays at 30-35 on every stack.

ROCmFP4 serving (measured on RX 7900 XTX / gfx1100)

cmake -B build-rocmfp4 -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1100
cmake --build build-rocmfp4 -j
bash scripts/serve-dflash2.sh

Complete launch command (what the script expands to)

LLAMA_ATTN_ROT_DISABLE=1 ./build-rocmfp4/bin/llama-server \
  -m  models/Qwen3.8-27B-heretic-ara.ROCmFP4-STRIX.gguf \   # main model (ROCmFP4/STRIX)
  -md models/Qwen3.8-27B-DFlash2-Q4_K_M.gguf \              # DFlash2 draft model
  --mmproj models/mmproj-Qwen3.8-27B-Q8_0.gguf \            # vision encoder (drop this line to disable)
  --spec-type ngram-map-k4v,draft-dflash \                  # stacked speculators, applied in order
  --spec-ngram-map-k4v-size-n 12 \                          # n-gram lookup key length
  --spec-ngram-map-k4v-size-m 48 \                          # max draft tokens on an n-gram hit
  --spec-ngram-map-k4v-min-hits 1 \                         # minimum occurrences for a hit
  --spec-draft-n-max 5 \                                    # DFlash2 draft depth (measured optimum)
  --spec-draft-p-min 0.4 \                                  # draft confidence cutoff (measured optimum)
  -ctk q4_0 -ctv q4_0 \                                     # 4-bit KV cache (~14 GiB saved @256K)
  -ngl 99 -ngld 99 \                                        # full offload: target / draft
  -c 262144 \                                               # 256K context
  -b 2048 -ub 512 \                                         # logical/physical batch, bounds prefill buffers
  -fa on \                                                  # fused attention kernels
  --jinja \                                                 # model's own chat template
  --no-kv-unified \                                         # prevents full-history re-prefill on hybrid archs
  --parallel 1 \                                            # single slot (pairs with the flag above)
  --ctx-checkpoints 2 \                                     # spec-decode rollback snapshots (default 32 ≈ 10 GiB)
  --cache-ram 4096 \                                        # spill prompt cache entries to host RAM
  --no-warmup \                                             # skip warm-up pass (VRAM is nearly full)
  --host 0.0.0.0 --port 1234                                # listen address and port

LLAMA_ATTN_ROT_DISABLE=1 is required only with a quantized KV cache (-ctk/-ctv not f16) on HIP: the hybrid-architecture cache path does not implement rotated KV and will crash without it. Do not set it for f16 KV.

Per-flag rationale and measured tuning sweeps: docs/rocmfp4.md.

Quick start

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models
• Release process

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
• sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain