HaloFPX

HaloFPX is a continuation-ready engineering monorepo for the custom dual-Strix- Halo inference project. The repository combines the C/C++ implementation lineage with the evidence-backed project wiki, research records, requirements, decisions, experiments, and handoff material used to govern that implementation.

Current target

HaloFPX is a llama.cpp-derived, model-architecture-general inference engine specialized for ROCmFPX-family GGUF model weights on the two AMD Strix Halo (gfx1151) target machines. Both physical targets are Nimo Direct MME3L systems running CachyOS; Ubuntu is a portability/vendor-support lane and the local Windows PC is the control environment. See the dated target-machine authority.

“Model-general” does not mean that the current optimization program targets every conventional GGUF quant equally. Primary performance work uses admitted ROCmFPX/ROCmFP4 artifacts across supported model architectures. Conventional quants are controls or fallbacks unless separately admitted. MiniMax Q6 Agent is the largest stress fixture, not the model-specific product target.

The current order is restart-safe saved prompt/KV-state reuse, measurable cache-path cost reduction, prompt-processing/TTFT improvement, and generation throughput/latency improvement. The living performance work plan records candidates and kill gates without turning them into claims.

Current work snapshot — 2026-08-12

• [VERIFIED] The cache-saving behavior reference is pinned to fewtarius/CachyLLama@6be745998f568e379ea197fcf827baec73ff9940. It is a behavior/reference authority, not evidence that its implementation or complete feature set has been imported.
• [VERIFIED] PR #23, merged as aee627bd46de21327c9082f7915818430d38f453, closed issue #14 with a default-off Linux CPU qualification of exact-key reuse across fresh server processes. Its scope is world size 1, rank 0, an ordinary transformer, and greedy memoryless sampling. Compatibility mismatch and same-size corruption miss and cold-recompute; this is not distributed HaloKV or prefix reuse.
• [VERIFIED] PR #27, merged as bf420e9f1db4ea4ba1d7c87771b6a4d662b5be67, added an optional, default-off OpenSSL EVP SHA-256 provider to the separate run-local SSD prompt cache. It retains exact-length and full-file digest checks. No target end-to-end cache-speed claim exists yet.
• [VERIFIED] PR #30 merged as 7a36e01a25bd5c27b684b489d9996b4de3afa299. It adds the first default-off ROCmFPX generation-path specialization: exact Q2/Q3/Q6/Q8 MMVQ consumers skip an unused Q8_1 activation-sum reduction. Both CachyOS Strix nodes compiled feature-on and feature-off source without restarting production; GPU correctness and matched performance remain open under issue #25.
• [VERIFIED] PR #35 merged as 167df62ffc8970bc408d72e97ab71a57de4b69d2 and fixed a mixed sampled/raw logits row-count crash. Issue #28 remains open for the coherent output snapshot and redundant-synchronization reduction.
• [OPEN] Prompt profiling, CachyOS A/B execution, cache metrics, two-rank cache composition, verified longest-prefix reuse, live-derived compatibility, sampling synchronization, and dense FFN conversion reuse are tracked in the work plan and GitHub issues #15, #18, #26, #28, #29, #32, #33, and #37. PR #31 closed #16 with the model-general frozen-plan/evidence core; #37 owns its CachyOS process adapter.
• [MEASURED] The 2026-08-12T23:06:08Z zero-restart health receipt is now historical before-state. A later target HMM/global-OOM incident restarted both ranks. The recovered nimo-1 coordinator was PID 3113343, InvocationID 0656332b63a140eab7214627baa43253, NRestarts=1; the recovered nimo-2 worker was PID 2248760, InvocationID d15fe49610274e77bd9a3d84a0b791a5, NRestarts=1. Both units were active/running, coordinator health was OK, and a real 5-prompt-token plus 1-generated-token request completed. The incident and recovery receipt supersedes only current service identity and recovery state; it is not a binary-hash refresh, benchmark, or performance result. Issue #41 is the P0 prerequisite for any target build, quantization, disposable inference, or benchmark.
• [VERIFIED] Hosted only: the offline maintenance admission controller now models issue-#41 shutdown, adapter cleanup, worker-first recovery, exact owner census, and the mandatory minimal two-rank probe through deterministic fakes. Its target gate is literal false and the controller defines or constructs no target Runner or SSH path. It is not a maintenance authorization or performance result.

Start here

• Worker start sequence — required reading and validation order before changing project material.
• Project handoff — exact source state, current milestone, validation commands, and the next safe work boundary.
• Artifact inventory — what is in Git, what is preserved as a release artifact, what is reproducible, and what is not available.
• License and provenance boundary — which implementation material is MIT and why the combined private preservation set is not covered by one blanket license.
• Canonical HaloFPX wiki — the engineering knowledge base; this in-repository copy is authoritative, not the optional GitHub Wiki surface.
• Publication-era project state and current lead decisions — preserved historical state plus the active decision authority and the evidence-backed decision record.
• Target machines — current CachyOS, hardware, role, access, production, and fresh-PC continuation authority.
• Performance work plan — current cache, prompt-processing, and generation lanes.

Current status and evidence boundary

The L111 work is a bounded loader-foundation milestone. It is accepted only when the publication reconciliation record is present in the lead decision log; without that record, treat it as a candidate with independent PASS evidence. Even when accepted, L111 does not establish production readiness, end-to-end runtime correctness, distributed-serving readiness, model-quality equivalence, or a current performance result.

Claims in project material retain their literal labels: [MEASURED] is tied to the recorded machine, software, model, and run conditions; [VERIFIED] requires primary evidence; [INFERENCE], [ASSUMPTION], and [RECOMMENDATION] are not facts; [OPEN] remains unresolved. The performance figures below are preserved historical measurements from 2026-07-12, not promises for this branch or other systems. Reproduce them from their evidence and matched configurations before using them for an engineering decision.

Implementation lineage: ROCmFPX for llama.cpp

ROCmFPX adds experimental AMD-focused 2-, 3-, 4-, 6-, and 8-bit GGUF model-weight formats to llama.cpp, with CPU reference paths and accelerated CUDA/HIP and Vulkan kernels where implemented. The current Q2 path has CPU plus selected CUDA/HIP operations and no Vulkan path; do not infer uniform backend coverage from family membership.

Implementation status: this source lineage is experimental. APIs, tuning choices, compatibility, and performance can change. The inherited guide below documents previously exercised paths; it is not a fresh validation of the publication branch. Results depend on hardware, drivers, model, prompt, and quantization recipe; use BF16/F16 sources for quality comparisons.

Why ROCmFPX?

• AMD-first weight formats: ROCmFP2, ROCmFP3, ROCmFP4, ROCmFP6, and ROCmFP8 are real GGUF model-weight quants, not just K/V-cache compression.
• Native accelerated paths: HIP/ROCm and Vulkan kernels are backed by CPU reference implementations for correctness testing.
• Speed and size choices: ROCmFP4 is the speed-first 4-bit family; existing Qwen comparisons put its files about 12% below the matched Q4_K_M size.
• Agent-aware presets: coherent/agent recipes protect tensors that matter for code, JSON, tool calling, and structured output.
• Built-in MTP acceleration: models with an MTP/NextN head—including M-RoPE Qwen models—can use target-verified self-speculative decoding without loading a separate draft model.
• Preserved validation history: the source lineage was exercised through local CPU/Vulkan/ROCm tests and cross-platform CI. Consult the linked evidence before relying on those historical results in a new environment.

Start with Quick Start, choose a format in Which Format Should I Pick?, or jump directly to MTP Speculative Decoding.

Historical measured MTP results — Strix Halo, 2026-07-12

These are [MEASURED] local command-line decode results from the source under test on Strix Halo (gfx1151). Throughput is the final Generation: rate from llama-cli; runs used full GPU offload, FlashAttention, -c 4096, greedy sampling (--temp 0), -b 512 -ub 512, the same prompt within each row, and one model at a time.

Qwable is a matched single 64-token pair. Qwen no-MTP is one 256-token run; the MTP values are the median and peak of three matched 256-token runs.

The source under test and the earlier experimental build were effectively tied on Qwen3.6: median differences were -0.7% on Vulkan and -0.3% on ROCm. On a longer 512-token Vulkan run, the tested source reached 110.7 t/s versus 107.2 t/s for the experimental build. Qwable's 256-token branch comparison was also tied: tested/experimental measured 32.9/33.0 t/s on Vulkan and 32.3/32.3 t/s on ROCm.

MTP gains are content-dependent: predictable code, JSON, and lists usually accept more draft tokens than creative prose. Treat the profiles above as tested starting points, not universal defaults.

Quick Start (Strix Halo / gfx1151)

This is the inherited operator recipe, retained so it can be reproduced and revalidated. It is not evidence that the publication branch currently builds or runs on a particular machine. For other AMD GPUs, select the corresponding build script from the Clone And Build table.

# 1. Get the code (HaloFPX monorepo main branch)
git clone https://github.com/JCFrags/HaloFPX.git
cd HaloFPX && git checkout main

# 2. Build for Strix Halo
env JOBS=16 scripts/build-strix-rocmfp4-mtp.sh          # -> build-strix-rocmfp4/

# 3. Quantize a BF16/F16 GGUF to ROCmFP4 (4.25 bpw, speed-first layout)
build-strix-rocmfp4/bin/llama-quantize model-BF16.gguf model-ROCMFP4_FAST.gguf Q4_0_ROCMFP4_FAST

# 4. Run it with the fastest backend measured on this Strix Halo system
build-strix-rocmfp4/bin/llama-cli \
  -m model-ROCMFP4_FAST.gguf -dev Vulkan0 -ngl 999 -fa on --jinja

That is the whole loop: build → quantize → run. The sections below explain each format, how to convert an existing NVFP4 model, and how to squeeze more decode speed with speculative decoding.

For a model that contains an MTP/NextN head, add a tested starting profile:

build-strix-rocmfp4/bin/llama-cli \
  -m model-with-MTP.gguf -dev Vulkan0 -ngl 999 -fa on --jinja \
  --temp 0 --spec-type draft-mtp \
  --spec-draft-n-max 6 --spec-draft-p-min 0.6

For Qwen3.6-35B-A3B on the tested Strix Halo system, n4 / p0.55 was faster:

--spec-draft-n-max 4 --spec-draft-p-min 0.55

To use HIP/ROCm instead of Vulkan:

export HSA_OVERRIDE_GFX_VERSION=11.5.1
export GGML_HIP_ENABLE_UNIFIED_MEMORY=1
build-strix-rocmfp4/bin/llama-cli \
  -m model-ROCMFP4_FAST.gguf -dev ROCm0 -ngl 999 -fa on --jinja

Historically tested support

Which Format Should I Pick?

Rule of thumb: start with Q4_0_ROCMFP4_FAST for speed, or a *_COHERENT / *_AGENT preset if the model does tool-calling, JSON, or coding. Always compare against your BF16/F16 source for real quality checks.

What Is ROCmFPX?

ROCmFPX is a family of GGUF model-weight quants:

Agent-specific versions are also available:

ROCmFPX is not a K/V-cache-only compression trick. It is a set of actual GGUF model-weight tensor formats. Q3/Q4/Q4_FAST/Q6/Q8 have CPU, CUDA/HIP, and Vulkan paths; Q2 currently has CPU plus selected CUDA/HIP operations and no Vulkan path.

Contributors And Credit

This work builds on llama.cpp; upstream authors and contributors retain credit under the MIT license. See AUTHORS, LICENSE, and THIRD_PARTY_NOTICES.md.

The imported ROCmFP4 and ROCmFPX experiment lineage was maintained by charlie12345 / caf; that attribution is retained in this monorepo.

Additional ROCmFPX contributors:

• ciru-ai: ROCmFPX FP3 Vulkan matvec/dequant speed path.
• Tom Turney / PlunderStruck / Aydan S.: TurboQuant turbo3/turbo4 K/V-cache quantization paths for ROCm/HIP and Vulkan.

Why It Is Different From Regular Quants

Most regular GGUF quants target broad size/quality tradeoffs. ROCmFPX is AMD-oriented and keeps the ROCmFP4 discipline:

• 32-weight blocks with format-specific CPU, HIP, and Vulkan coverage
• finite unsigned UE4M3 scale bytes
• explicit integer-code-times-decoded-scale dequant math
• reconstruction-MSE scale selection where low-bit coherency needs it
• tensor-aware routing for low-bit coherency instead of applying one blunt type everywhere
• optional agent presets for JSON, tool calling, coding, and chat coherency

The agent presets do not invent a separate dequant kernel. They use the same ROCmFPX math but protect the tensors that tend to break structured output: token/output embeddings, attention Q/K/V/O, selected FFN-down, and selected FFN-gate tensors.

Detailed and Historical Benchmarks

These are additional pre-promotion comparisons from a Strix Halo / gfx1151 system. Treat them as historical local data, not a universal benchmark. All rows within each table used the same model pair, backend, batch shape, K/V cache, FlashAttention setting, and one test at a time.

Qwen3.6 27B, Vanilla No-MTP

Model pair:

• Baseline: Qwen3.6-27B-Q4_K_M.gguf, 16.55 GB
• ROCmFPX: Qwen3.6-27B-VANILLA-NO-MTP-BF16-to-ROCmFP4-STRIX_LEAN.gguf, 14.59 GB
• Size delta: ROCmFP4 is 11.82% smaller
• Test: llama-bench, pp512 + tg128, MTP/speculative decoding disabled

On this 27B vanilla run, ROCmFP4 was slightly behind Q4_K_M for ROCm prompt fill, but faster for decode on both ROCm and Vulkan. Vulkan ROCmFP4 also led prompt fill.

Qwen3.6 35B A3B Weight-Quant Comparison

Model pair:

• Baseline: Qwen3.6-35B-A3B-MTP-Q4_K_M.gguf, 21.71 GB
• ROCmFPX: Qwen3.6-35B-A3B-MTP-BF16-to-ROCmFP4-STRIX_LEAN-ROCmFPXCLONE.gguf, 19.05 GB
• Size delta: ROCmFP4 is 12.28% smaller
• Test: llama-bench, pp512 + tg128; this table measures the weight quants, not speculative MTP acceleration

The same 35B A3B pair was also run through a 20-prompt Hermes-style agent smoke:

On this 35B A3B comparison, ROCmFP4 was smaller and faster on decode/generation across ROCm and Vulkan. ROCm prompt fill was still slightly behind Q4_K_M in llama-bench, while Vulkan prompt fill and Hermes-style prompts favored ROCmFP4.

Clone And Build

git clone https://github.com/JCFrags/HaloFPX.git
cd HaloFPX

Use main for this monorepo. Historical donor and pre-publication source lineages are checksum-bound in private release evidence-2026-08-12 and described in ARTIFACTS.md; no publication branch named experimental-rocmfpx-branch is asserted here.

Pick the build script for your machine:

For Strix Halo, the common runtime environment is:

export HSA_OVERRIDE_GFX_VERSION=11.5.1
export GGML_HIP_ENABLE_UNIFIED_MEMORY=1

Key binaries after build:

build-strix-rocmfp4/bin/llama-quantize
build-strix-rocmfp4/bin/llama-cli
build-strix-rocmfp4/bin/llama-server
build-strix-rocmfp4/bin/llama-bench
build-strix-rocmfp4/bin/test-backend-ops

For RDNA2/RDNA3/RDNA4 builds, use the same binary names under that build folder, for example build-rdna3/bin/llama-quantize.

The build-rocmfp4-rocm714-local.sh script (RDNA4 / RX 9000) selects the ROCm 7.14.0a20260624 nightly by versioned name, but the nightly URL is not an immutable byte identity. The script therefore requires an independently recorded ROCM_TARBALL_SHA256 before it downloads or extracts the toolchain. It also requires patchelf, copies candidate runtime libraries, and fails if ldd reports an unresolved direct dependency or resolves a recognized ROCm dependency outside that candidate directory. This is still only a build-host packaging check: the script does not prove that every data package (including device-code payloads) is relocatable. Run a clean-host, target-GPU smoke test before describing the folder as portable or distributable.

Quantize Straight ROCmFPX Models

Use BF16 or F16 GGUF sources. The wrapper keeps split GGUFs split by default.

ROCmFP2 is currently a direct-quantizer-only experimental path; the wrapper does not expose a Q2 or agent preset:

build-strix-rocmfp4/bin/llama-quantize source-BF16.gguf \
  out-Q2_0_ROCMFPX.gguf Q2_0_ROCMFPX

ROCmFP3:

SRC=/path/to/model-BF16.gguf OUT=/path/to/model-Q3_0_ROCMFPX.gguf \
  FORMAT=rocmfp3 PROFILE=straight scripts/quantize-rocmfpx-agent.sh

ROCmFP4:

SRC=/path/to/model-BF16.gguf OUT=/path/to/model-Q4_0_ROCMFP4.gguf \
  FORMAT=rocmfp4 PROFILE=straight scripts/quantize-rocmfpx-agent.sh

ROCmFP6:

SRC=/path/to/model-BF16.gguf OUT=/path/to/model-Q6_0_ROCMFPX.gguf \
  FORMAT=rocmfp6 PROFILE=straight scripts/quantize-rocmfpx-agent.sh

ROCmFP8:

SRC=/path/to/model-BF16.gguf OUT=/path/to/model-Q8_0_ROCMFPX.gguf \
  FORMAT=rocmfp8 PROFILE=straight scripts/quantize-rocmfpx-agent.sh

You can also call llama-quantize directly:

build-strix-rocmfp4/bin/llama-quantize source.gguf out-q3.gguf Q3_0_ROCMFPX
build-strix-rocmfp4/bin/llama-quantize source.gguf out-q4.gguf Q4_0_ROCMFP4
build-strix-rocmfp4/bin/llama-quantize source.gguf out-q6.gguf Q6_0_ROCMFPX
build-strix-rocmfp4/bin/llama-quantize source.gguf out-q8.gguf Q8_0_ROCMFPX

For low-bit ROCmFPX quants, pass an imatrix when you have one:

IMATRIX=/path/to/imatrix.gguf \
SRC=/path/to/model-BF16.gguf OUT=/path/to/model-Q3_0_ROCMFPX.gguf \
  FORMAT=rocmfp3 PROFILE=straight scripts/quantize-rocmfpx-agent.sh

The wrapper forwards IMATRIX to llama-quantize --imatrix. ROCmFP3, ROCmFP6, and ROCmFP8 use imatrix-weighted scale search; ROCmFP4 has its own imatrix path.

Quantize Agent ROCmFPX Models

Use agent mode when the model will be used for Hermes/OpenClaw-style workflows, tool calling, JSON output, coding, or chat agents.

ROCmFP3 Agent:

SRC=/path/to/model-BF16.gguf OUT=/path/to/model-Q3_0_ROCMFPX_AGENT.gguf \
  FORMAT=rocmfp3 PROFILE=agent scripts/quantize-rocmfpx-agent.sh

ROCmFP6 Agent:

SRC=/path/to/model-BF16.gguf OUT=/path/to/model-Q6_0_ROCMFPX_AGENT.gguf \
  FORMAT=rocmfp6 PROFILE=agent scripts/quantize-rocmfpx-agent.sh

ROCmFP8 Agent:

SRC=/path/to/model-BF16.gguf OUT=/path/to/model-Q8_0_ROCMFPX_AGENT.gguf \
  FORMAT=rocmfp8 PROFILE=agent scripts/quantize-rocmfpx-agent.sh

ROCmFP4 Agent:

SRC=/path/to/model-BF16.gguf OUT=/path/to/model-Q4_0_ROCMFP4_COHERENT_AGENT.gguf \
  FORMAT=rocmfp4 PROFILE=agent scripts/quantize-rocmfpx-agent.sh

The wrapper maps FORMAT and PROFILE like this:

Convert An Existing NVFP4 Model To ROCmFP4

If you already have an NVFP4 GGUF, you can re-map it onto the ROCmFP4 kernel path without re-quantizing from BF16. This is the closest-matching conversion ROCmFPX supports: NVFP4 and ROCmFP4 use the same UE4M3 scale and share 7 of 8 codebook levels — only the top magnitude level differs (NVFP4 12 vs ROCmFP4 10), so almost every weight maps over cleanly.

# Same 4.50 bpw as NVFP4 (closest quality match):
build-strix-rocmfp4/bin/llama-quantize --allow-requantize \
  model-NVFP4.gguf model-ROCMFP4.gguf Q4_0_ROCMFP4

# Smaller 4.25 bpw (speed-first layout, a little more loss):
build-strix-rocmfp4/bin/llama-quantize --allow-requantize \
  model-NVFP4.gguf model-ROCMFP4_FAST.gguf Q4_0_ROCMFP4_FAST

• --allow-requantize is required: NVFP4 GGUFs usually keep output.weight at a higher-precision type (e.g. q6_K), so the file has mixed source types.
• Example measured on a 9B NVFP4 model (wikitext-2, gfx1151): the 4.50 bpw target landed within noise of the NVFP4 source perplexity; the 4.25 bpw FAST target was ~5% higher perplexity for a ~10% smaller file. Numbers are model-dependent — always A/B against the NVFP4 source on your own prompts.
• To make every tensor ROCmFP4 (a uniform "even" file), use the Q4_0_ROCMFP4_EVEN / Q4_0_ROCMFP4_FAST_EVEN presets, which imply --pure.

Faster Decode: MTP Speculative Decoding

If your model ships with an MTP / NextN draft head (many recent models do), you can turn on self-speculative decoding for a real decode speedup — no separate draft model needed. This is the most effective way to push decode throughput past what the weight format alone can do, because accepted draft tokens produce several tokens per weight read.

MTP helps both dense and MoE models here. On gfx1151, -dev Vulkan0 was the fastest backend in the validated Qwen3.6 and Qwable comparisons.

# General starting profile for a model with an embedded MTP/NextN head
build-strix-rocmfp4/bin/llama-cli \
  -m model-with-MTP.gguf -dev Vulkan0 -ngl 999 -fa on --jinja \
  --temp 0 \
  --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.6

• Tune per model and workload. n6 / p0.60 is a useful starting point, but the validated Qwen3.6-35B-A3B profile was faster at n4 / p0.55. Very low p_min can waste work on rejected drafts; an overly high value can miss useful draft tokens.
• The speedup is content-dependent: structured / predictable output (code, lists, JSON) accepts more drafts and gains most; free-form creative text gains less.
• It is lossless: at greedy (--temp 0) the output matches non-speculative decoding token-for-token (the target model verifies every drafted token).
• See Historical measured MTP results for the dated Qwable and Qwen3.6 measurements, profiles, and branch-parity context.

M-RoPE models (qwen35 / qwen35moe, and any IMROPE/MROPE arch): MTP now works on these. They use 4-D M-RoPE positions, and the batch position check previously rejected the MTP draft/verify batch every step (for M-RoPE, it is required that the position satisfies: X < Y), so MTP silently fell back to plain decode. The MTP hook batch is a hybrid (token id plus an injected hidden-state row) and is allowed to reuse positions like an embedding batch, so the strict check is now gated on batch.token && !batch.embd (src/llama-batch.cpp). If you are on an older build and see that X < Y error spamming during MTP, this is the fix. NEOX-RoPE MTP (e.g. Gemma4 assistants) was never affected.

What The Agent Preset Protects

The agent profile is a tensor-routing choice. It keeps the ROCmFPX block formats but spends more bits on tensors that affect structured behavior:

• token and output embeddings
• attention Q/K/V/O tensors
• selected FFN-down tensors
• selective FFN-gate tensors
• bulk FFN-up tensors stay on the family quant where possible

This is why agent quants are slightly larger than straight quants. The goal is to preserve JSON shape, tool-call shape, coding behavior, and chat coherency without forcing the whole model to a generic high-bit quant.

Run A Quantized Model

Simple ROCm run:

build-strix-rocmfp4/bin/llama-cli \
  -m /path/to/model-Q8_0_ROCMFPX_AGENT.gguf \
  -dev ROCm0 \
  -ngl 999 \
  -fa on \
  -c 8192 \
  -b 512 \
  -ub 512 \
  --jinja

OpenAI-compatible server:

build-strix-rocmfp4/bin/llama-server \
  -m /path/to/model-Q8_0_ROCMFPX_AGENT.gguf \
  --host 127.0.0.1 \
  --port 8138 \
  -dev ROCm0 \
  -ngl 999 \
  -fa on \
  -c 8192 \
  -b 512 \
  -ub 512 \
  --jinja \
  --reasoning off

K/V Cache Rule

ROCmFPX model quants and K/V cache types are separate runtime controls.

The current guard promotes -ctk q3_0_rocmfpx to q6_0_rocmfpx because fp3 K cache was below the observed tool-call and agent coherency floor. q3_0_rocmfpx can still be used for V cache.

TurboQuant K/V cache support is already built into this tree as the turbo3 and turbo4 runtime cache types, including CPU reference tests plus ROCm/HIP and Vulkan paths. TurboQuant is not a ROCmFPX model-weight quant; use it with -ctk and -ctv at runtime.

The recommended safe TurboQuant+ style policy is asymmetric K/V:

build-strix-rocmfp4/bin/llama-server \
  -m /path/to/model-Q6_0_ROCMFPX_AGENT.gguf \
  -dev Vulkan0 \
  -ngl 999 \
  -fa on \
  -ctk q8_0 \
  -ctv turbo4 \
  --jinja

For the ROCmFPX MTP server wrapper, use the preset script:

MODEL=/path/to/model-Q6_0_ROCMFPX_AGENT.gguf \
DEVICE=Vulkan0 \
scripts/run-rocmfpx-turboquant-asym-server.sh

This keeps K cache at q8_0, where attention quality and tool calls are more sensitive, and uses turbo4 for V cache, where compression is usually cheaper. You can still run symmetric TurboQuant for sweeps with -ctk turbo3 -ctv turbo3 or -ctk turbo4 -ctv turbo4, but do not treat those as the default agentic serving profile.

For symmetric TurboQuant experiments, first/last-layer K protection is available as an opt-in compatibility knob:

LLAMA_KV_TURBO_BOUNDARY_LAYERS=2 \
build-strix-rocmfp4/bin/llama-server \
  -m /path/to/model.gguf \
  -ctk turbo4 \
  -ctv turbo4

With that flag, the first and last two model layers use q8_0 for K cache while the middle layers use the requested TurboQuant type. V boundary protection is off by default; enable it only for experiments with LLAMA_KV_TURBO_BOUNDARY_V=1.

Do not import the Python turboquant_plus research package into this C/C++ tree as-is. The low-risk production findings are the asymmetric K/V policy and documentation. QJL and turbo2 are intentionally not enabled here, and block-size 128 would require a GGML block-layout change and compatibility work.

Test Agent Behavior

The agentic smoke harness checks chat, coding, JSON, tool-call JSON, coherency, and streaming. It also refuses to start when ROCm reports an active KFD process, so each run starts after VRAM/process cleanup.

MODEL=/path/to/model-Q8_0_ROCMFPX_AGENT.gguf \
BACKEND=ROCm0 \
ALIAS=rocmfpx-agent \
OUT_DIR=/tmp/rocmfpx-agentic-smoke \
scripts/check-rocmfpx-agentic-smoke.sh

Code Layout

• ggml/rocmfpx/ - ROCmFP2/ROCmFP3/ROCmFP6/ROCmFP8 reference formats
• ggml/rocmfp4/ - ROCmFP4 reference path this family inherits from
• scripts/quantize-rocmfpx-agent.sh - simple straight-vs-agent quant wrapper
• scripts/check-rocmfpx-agentic-smoke.sh - OpenAI-compatible agent smoke test
• docs/ROCmFPX-HANDOFF.md - detailed handoff for reviewers and other agents
• docs/ROCmFPX-EXPERIMENT.md - experiment history, routing notes, and gates
• docs/BUILD-AMD-ARCHITECTURES.md - RDNA2/RDNA3/RDNA4/Strix build details

License

The implementation lineage is based on llama.cpp and retains the upstream MIT license where it applies. This combined repository also preserves research, evidence, tools, and third-party material under other or unresolved terms; the root LICENSE is not a blanket license for those materials. See LICENSES_AND_PROVENANCE.md, LICENSE, and THIRD_PARTY_NOTICES.md before copying or redistributing any subset.