llama.cpp: fork with multi gpu acceleration even for models bigger than total gpu mem

Big thanks to @csantiago78: the expert cache here builds on their implementation in llama.cpp PR #27861 ("GPU-resident LRU cache for host-offloaded MoE expert weights"), the first to get a working hot-expert cache into llama.cpp. This fork takes it further: filling all free VRAM, smarter eviction, CPU/GPU overlap and fused kernels.

For MoE models much larger than VRAM: every expert stays in system RAM, and all VRAM left after the KV cache becomes a live cache of the experts actually being used. The GPUs compute cached experts while the CPU computes the rest, in parallel.

Results, 2x RTX 3090 (48 GB) + 125 GB RAM, single stream, temp 0, -c 1024:

GLM prompt: "generate smallest html tetris game."; MiMo/Qwen prompt: "write smallest html tetris game" (both temp 0). PPL: wikitext-2, 40 x 512-token chunks (GLM only). MiMo needed -fitt 8000 on both stock and fork to avoid autofit OOM-ing on this arch/quant combo; the others loaded fine with default fit.

Qwen's gain is small because stock's autofit already placed most of its experts on GPU by default here — little room left for the cache to improve on. The big wins (GLM, MiMo) are on models where default placement leaves most expert work on the CPU.

Models:

• GLM original GGUF (tested): pfeifferj/GLM-5.3-Flash-GSQ-RCO-GGUF, the 3.0-bit file. It works as-is with this fork, no conversion needed, and it's the quality reference (unchanged perplexity). Most of the speedup comes from the fork, not the requantization.
• GLM Q4_K attention variant (same experts, non-expert Q8_0 weights requantized to Q4_K): neuralll/GLM-5.3-Flash-GSQ-RCO-3.0bit-Q4Kattn-GGUF.

Same VRAM, different use. Stock llama.cpp and this fork get the same 48 GB; what differs is what it holds. Each token uses only 8 of the 288 experts in each layer.

• Stock places experts statically, whole layers at a time: ~36 GB fits all 288 experts of ~14 of the 42 MoE layers. Most of that VRAM holds experts the current token doesn't touch, so only ~33% of each token's expert work runs on GPU and the CPU does ~67%, one after the other.
• This fork fills the same VRAM with the ~100 most-used experts of every layer. Usage is skewed, so those cover ~85% of what tokens actually pick: ~85% of expert work runs on GPU and the CPU does ~15%, at the same time as the GPUs.

So stock can't reach 2x on the same hardware: without an expert cache, extra VRAM mostly holds experts that aren't being used.

Run (with the faster Q4_K attention GGUF from neuralll/GLM-5.3-Flash-GSQ-RCO-3.0bit-Q4Kattn-GGUF):

llama-server -m GLM-5.3-Flash-GSQ-RCO-3.0bit-q4kattn.gguf \
    -np 1 -c 1024 -t 6 --cpu-moe -nr --moe-expert-cache -1

• --cpu-moe keeps all experts in RAM; --moe-expert-cache -1 sizes the cache per GPU from the VRAM free after KV and compute buffers. Set -c explicitly: without it autofit grows the context and takes the VRAM the cache needs.
• -t 6 suits an 8-core CPU (leave cores to drive the GPUs).
• Cache hit rate is ~85% after warm-up. LLAMA_MOE_CACHE_STATS=1 logs it.
• Tuning: LLAMA_MOE_CACHE_POLICY (add default, halve, window), LLAMA_MOE_CACHE_MARGIN_MB (VRAM left free, default 1024), LLAMA_MOE_CACHE_SWAP_FRAC (share of token time for uploads, default 0.25).
• GGML_SCHED_PROF=1 prints where each token's host time goes.

More GPUs (estimate, only 2 tested). Nothing assumes two GPUs: each GPU gets its own cache, sized from its free VRAM. Each extra GPU adds cache room, so more of every token's experts are hits and less work falls to the CPU. For this model, per token today: ~20 ms GPU work on non-expert layers, ~10 ms CPU on missed experts.

The plateau is the ~20 ms GPU part: with the default layer split each layer runs on one GPU at a time, so extra GPUs add cache room, not speed on that part. System RAM must still hold all experts. Cards in x4 PCIe slots upload experts slower, so the cache warms up slower. Reports from 3+ GPU setups are welcome.

What's in it: the GPU expert cache from PR #27861 (csantiago78), extended with VRAM-filling auto-sizing, usage-driven eviction that only swaps when the upload pays back, CPU/GPU overlap per layer, scheduler barrier fixes and fused gate kernels. GLM-5.3-Flash support comes from PRs #27773 and #27917 (timkhronos); stock llama.cpp can't load GLM-5.3-Flash yet.

Prefill warm start, implemented independently for this fork: the cache observes which experts the prompt itself selects during prefill and preloads them before the first generated token, instead of starting cold and only learning from decode. @sdroege explored the same idea independently in the PR #27861 discussion with their own patch; worth checking out too.

Other notable tweaks:

• Swap budget from measured cost, not a guess: each step measures real upload time (ms/expert) and real token time, then computes how many swaps fit in LLAMA_MOE_CACHE_SWAP_FRAC of a token (default 25%) — instead of a fixed swaps-per-step constant.
• Pay-back filter on every eviction: a swap only happens if the candidate's measured usage beats the cached victim's by more than what the upload itself costs in CPU-equivalent time, so churn can't cost more than it saves.
• Usage tracking with decay (LLAMA_MOE_CACHE_POLICY: add default, halve, window): recent use counts more than old use, so the cache follows shifts in which experts are hot instead of freezing on early-token bias.
• CPU/GPU overlap inside a layer: the GPU cache chain is queued and its inputs copied before the CPU miss chain runs, so both compute at the same time instead of the scheduler serializing them.
• Scheduler fixes upstream benefits from too: no host barrier between two GPU splits when neither reads host memory, and a new split is inserted exactly when another GPU's result is needed mid-split — both apply to any multi-GPU llama.cpp workload, not just this cache.
• Fused CUDA kernel for the hyper-connection/KDA gate chain (MUL -> ADD|SCALE -> SIGMOID -> SCALE, one kernel instead of four), and GLM5-Next's KDA Q/K norm collapsed from rms_norm+scale into one l2_norm op.
• Repacked CPU experts stay cacheable: uploads read raw bytes straight from the GGUF file (recorded per-tensor file offsets) when host memory holds a repack-transformed layout instead of the on-disk one.
• GGML_SCHED_PROF=1 and LLAMA_MOE_CACHE_STATS=1 for live profiling: barrier vs. copy wait time, fill %, in-flight uploads, queue depth, hit rate.

About the author of this fork: I'm actively looking for an AI engineering/research role and open to relocating out of Eastern Europe. If this work is useful to you or your team, reach out: linkedin.com/in/neuralll

---

Quick start

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models
• Release process

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
• sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain