GLM-5.3-Flash GSQ-RCO 3.0-bit, Q4_K attention (GGUF)

Big thanks to @csantiago78: the expert cache here builds on their implementation in llama.cpp PR #27861 ("GPU-resident LRU cache for host-offloaded MoE expert weights"), the first to get a working hot-expert cache into llama.cpp. This fork takes it further: filling all free VRAM, smarter eviction, CPU/GPU overlap and fused kernels.

A variant of pfeifferj/GLM-5.3-Flash-GSQ-RCO-GGUF (3.0-bit) made for faster single-stream decoding of this 117 GB model on consumer GPUs with an expert cache.

Only the non-expert Q8_0 weights changed (attention, shared experts, dense FFN, output head: Q8_0 to Q4_K, ~8.3 GB of the file). All 126 routed-expert tensors, token_embd and every F32/BF16 tensor are copied bit-for-bit from the original. Those Q8_0 weights are streamed by the GPU for every token, so shrinking them speeds up decoding; the experts are unchanged.

As usual GLM is new and this model requires a llama.cpp fork

Stock llama.cpp can't load GLM-5.3-Flash yet. Use neurall/llama.cpp (GLM-5.3-Flash support plus a VRAM-filling MoE expert cache). It adds real MULTI-GPU acceleration: For examplle on my 2 GPUs, 2x the tokens per second of stock llama.cpp (12.3 to 26-28 t/s):

llama-server -m GLM-5.3-Flash-GSQ-RCO-3.0bit-q4kattn.gguf \
    -np 1 -c 1024 -t 6 --cpu-moe -nr --moe-expert-cache -1

Set -c explicitly (without it autofit grows the context and takes the VRAM the cache needs).

But this fork is different. It not only adds GLM support but adds speedup and better GPU scalability too: stock llama.cpp splits a model across GPUs by layer, so for a single request only one GPU works at a time: GPU1 idles while GPU2 runs its layers, and both idle while the CPU computes the majority ? of experts that didn't fit in VRAM are processed on CPU fallback. A second GPU adds memory, not parallel compute. This fork turns that memory into a live cache keeping only the most used experts within monitored window of 64 tokens, so most most active experts work actually runs on the GPUs, and in parallel with the CPU computing the rest. On 2x RTX 3090: 12.3 t/s stock vs 26-28 t/s with the fork, over 2x. The file also loads on any build with GLM-5.3-Flash support (PRs #27773 / #27917), at stock speed.

Same VRAM, but more efficient use. Stock llama.cpp and this fork get the same 48 GB; what differs is what it holds. Each token uses only 8 of the 288 experts loaded in VRAM in each layer.

• Stock places experts statically, whole layers at a time: ~36 GB fits all 288 experts of ~14 of the 42 MoE layers. Most of that VRAM holds all layers experts the current token often doesn't touch, so only ~33% of each token's expert work runs on GPU and the CPU does ~67%, one after the other.
• This fork fills the same VRAM with the ~100 most-used experts of every layer. Usage is skewed, so those cover ~85% of what tokens actually pick: ~85% of expert work runs on GPU and the CPU does ~15%, at the same time as the GPUs.

So stock can't reach 2x on the same hardware: without an expert cache, extra VRAM mostly holds again experts that aren't often being used by single prompt.

Results

2x RTX 3090 (48 GB VRAM) + 125 GB RAM, 8-core CPU, single stream. Decode: prompt "generate smallest html tetris game.", 1024 context, temperature 0. Perplexity: wikitext-2 test, 40 x 512-token chunks, same build for both files.

Note: the original quant's RCO allocation chose each tensor's precision on purpose; overriding the Q8_0 tensors to Q4_K trades ~1% perplexity for ~10% speed but we touched just attention. If you want the RCO allocation as designed, use the original file or any model for that matter. I chosen GSQ-RCO as base as I loved the rest tensors having better quants in better efficiency. Perhaps attn quant can be GSQ-RCO requantized as well.

How it was made

llama-quantize --allow-requantize --tensor-type-file tensor-types-q4kattn.txt \
    GLM-5.3-Flash-GSQ-RCO-3.0bit.gguf GLM-5.3-Flash-GSQ-RCO-3.0bit-q4kattn.gguf Q4_K

tensor-types-q4kattn.txt (in this repo) lists every tensor with its target type: Q8_0 non-expert tensors (except token_embd) to q4_k, everything else its original type. Tensors whose type doesn't change are copied, not requantized.

Credits and license

• Base model: zai-org/GLM-5.3-Flash, MIT.
• Original quantization: pfeifferj/GLM-5.3-Flash-GSQ-RCO-GGUF, a community reproduction of the GSQ and RCO methods by IST-DASLab (GSQ, RCO). Not an IST-DASLab release.
• Expert cache: builds on llama.cpp PR #27861 (csantiago78).
• GLM-5.3-Flash support: llama.cpp PRs #27773 and #27917 (timkhronos).

MIT license, see LICENSE.