NVIDIA-Nemotron-3.5-Lightning-30B-A3B — ROCmFP4-STRIX_LEAN

TL;DR

NVIDIA Nemotron 3.5 Lightning 30B-A3B (hybrid Mamba-2 + MoE + Attention) quantized to ROCmFP4-STRIX_LEAN (software FP4, GGUF type 106) for AMD Strix Halo (gfx1151) on the ROCmFPX fork family — we serve and benchmark these files on our lab runtime (full source: pugant/strix-nebulosa; upstream: charlie12345/ROCmFPX). Runs at 84.3 tok/s generation on a Radeon 8060S iGPU — the fastest of any model tested on this hardware. 15.7 GiB footprint.

⚠️ This GGUF is for the ROCmFPX fork of llama.cpp on AMD RDNA 3.5 only. It will not load in stock llama.cpp.

⚠️ Critical warnings — read before use

1. Type 106 is invalid for stock llama.cpp. Q4_0_ROCMFP4_STRIX_LEAN (type 106) is a custom type from the charlie12345/ROCmFPX fork. Loading this file with a stock build will fail.
2. Requires a ROCmFPX fork build — see Runtime. Upstream charlie12345/ROCmFPX loads these files too (any build with the custom GGML types, e.g. the matching kyuz0 Docker toolbox for gfx1151).
3. FP4 is software-dequantized on RDNA 3.5. gfx1151 has no native FP4 hardware; the kernel dequantizes FP4→FP16 in software. The speedup vs. Q4_K_M comes from the smaller memory footprint and the fork's optimized ROCm dequant path, not from hardware FP4.
4. Unified memory required. On Strix Halo the VRAM partition is tiny (512 MB). Run with GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 and HSA_OVERRIDE_GFX_VERSION=11.5.1.
5. -fit off is mandatory for this architecture. llama-server with default -fit on deadlocks during "fitting params to device memory" on the 128-expert MoE. Always pass -fit off.

Benchmarks

Hardware: AMD RYZEN AI MAX+ 395, Radeon 8060S (gfx1151, 128 GB unified memory), ROCm 7.2.4. Method: llama-bench -ngl 999 -fa on -p 512 -n 128. All benchmarks for this model were run in ROCm containers (HIP backend). Since then we also benchmarked ROCmFPX quants on the Vulkan (RADV) build of the fork — see the comparison table in Qwen3.6-35B-A3B-MTP-Q6_0_ROCMFPX; those numbers are on a different backend and not directly comparable.

ROCmFP4-STRIX_LEAN is +32% faster at generation and 31% smaller than Q4_K_M. Note: Q4_K_M falls back on 134/401 tensors (the Mamba/SSM tensors are not compatible with Q4_K_M block sizes); ROCmFP4 quantizes them natively (1/401 fallback), which is a large part of why it wins here.

Compared to other Strix Halo ROCmFP4 models: this is the fastest yet (+19% vs grug-35b-v2 at 70.92, +26% vs Ornith-1.0-35B at 66.68).

Quantization details

• Type: Q4_0_ROCMFP4_STRIX_LEAN (type 106) — fork-specific.
• Preset: STRIX_LEAN (uses q4_0_rocmfp4_fast dequant).
• Bitrate: 4.28 BPW.
• imatrix: bartowski calibration (185 entries covering the quantizable tensors).
• Tensors: 401 (237 f32 — norms/SSM-state/bias; 164 ROCmFP4 — weight matrices).
• SHA256: 91316f85d52d378ff963c42f293660f2ca094f8bce0b08ea481ff620bc5056a7

Architecture: nemotron_h_moe

Nemotron-H is a hybrid SSM-Transformer. Each of the 52 layers is one of:

• Mamba-2 (SSM): linear-time recurrent state-space layer. ssm.conv_kernel=4, ssm.state_size=128, 64 heads. Handles long-range context with fixed KV cost.
• MoE: 128 routed experts (6 active per token) + 1 shared expert. expert_feed_forward_length=1856, expert_shared_feed_forward_length=3712.
• Multi-head attention: 32 heads, 2 KV heads (GQA), head_dim=128, RoPE θ=10000. Inserted periodically for global mixing.

Stats: 31.58 B total parameters, ~3.5 B active per token, 131072 vocab, 1M context length.

Note: the ROCmFPX fork's converter does not emit the MTP/nextn tensors, so this build is a plain (non-speculative) model. The base model's MTP layer is not used here.

imatrix

The importance matrix from bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF (185 entries, 822 chunks). Paths in the GGUF metadata have been stripped to basenames during sanitization.

Usage

docker run --rm -d --name nemotron \
  --device /dev/kfd --device /dev/dri --group-add video --group-add render \
  -e HSA_OVERRIDE_GFX_VERSION=11.5.1 -e GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 \
  -p 1234:1234 -v /path/to/models:/models \
  <rocmfpx-fork-image> llama-server \
    -m /models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN.gguf \
    -ngl 999 -c 32768 -fa on --jinja --host 0.0.0.0 --port 1234 -fit off

The model has a chat template with a thinking/reasoning mode (the <think> block surfaces in reasoning_content on the OpenAI-compatible endpoint).

Attribution

• Base model: NVIDIA, nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (OpenMDW-1.1).
• Converter & quantization: charlie12345/ROCmFPX fork (MIT).
• imatrix: bartowski.
• Toolbox: kyuz0 llama.cpp-amd-gpu.
• Quantized by: pugant.

License

OpenMDW License Agreement, version 1.1 (OpenMDW-1.1) — inherited from the upstream NVIDIA model. See LICENSE.

Acknowledgements

Thanks to NVIDIA for releasing Nemotron under a permissive license, to the ROCmFPX fork authors for the FP4 quantization work, and to bartowski for the calibration imatrix.

Limitations

• FP4 dequantization is software-only on gfx1151; no native FP4 hardware is used.
• MTP/speculative decoding is not enabled in this build.
• Validated on Strix Halo (gfx1151) only; behavior on other RDNA versions is untested.

Disclaimer

Not affiliated with or endorsed by NVIDIA, AMD, ggml-org, bartowski, kyuz0, or the ROCmFPX maintainers. Provided "as is", without warranty.

---

Runtime

Benchmark environment: one bare-metal AMD Strix Halo (Ryzen AI MAX+ 395, 128 GB) — full dated configuration and measurement policy: BARE-METAL.md

Requires a ROCmFPX fork build (custom tensor types — stock llama.cpp refuses the file). Recommended: our lab build (pugant/strix-nebulosa, main) — reasoning budget and persistent prompt cache on every model; drafter features where the model ships one: see the engine section of its README.

The first Mamba-hybrid through the preset; served and benchmarked on our lab build.

Everything here is experimental and provided as-is, at your own risk.