llama.cpp

⚡ This fork — Fable's MoE-offload prefill optimizations

Two opt-in optimizations for large MoE models whose experts are offloaded to system RAM (--n-cpu-moe), found and implemented by Fable. Both are off by default, toggled via environment variables, and produce token-identical output to mainline.

Benchmark

Measured on an RTX 3060 12GB with Qwen3.6-35B-A3B (--n-cpu-moe 26), prompt-processing at 2048 (MODEL = path to your .gguf):

# baseline (patches off):
./build/bin/llama-bench -m MODEL -ngl 99 -ncmoe 26 -p 2048 -n 0 -r 5 -b 2048 -ub 2048

# patched (both optimizations on):
GGML_CUDA_REGISTER_HOST=1 GGML_SCHED_PREFETCH_EXPERTS=1 \
./build/bin/llama-bench -m MODEL -ngl 99 -ncmoe 26 -p 2048 -n 0 -r 5 -b 2048 -ub 2048

Result: ~1143 → ~1880 t/s prefill (+64%) — same GPU, same settings, token-identical.

Branches: fable5/host-register (pinning only) · fable5/prefetch-experts (both — this branch).

⚡ This fork — MoE expert cache (VRAM-resident hot experts)

For MoE models whose routed experts live in system RAM (--n-cpu-moe), this fork can keep the most-frequently-routed experts of each layer resident in VRAM. Decode runs the hot experts on GPU and only the cold remainder on CPU; the two halves are merged exactly, so output is bit-identical to baseline. Opt-in, off by default.

Measured on an RTX 3060 12GB (-ngl 99 -ncmoe 99 -fa 1):

Supported architectures: qwen35moe, qwen4exp (Qwen3.8-Flash-Next), deepseek2, laguna (plain fused-SILU gated expert FFN, separate gate/up/down tensors). Other architectures run unchanged.

Quick start

1. Capture a routing profile (one time per model — records which experts the router picks):

MOE_TRACE_OUT=mymodel-code.csv ./build/bin/llama-moe-trace -m model.gguf \
  -ngl 99 -ncmoe 99 -fa 1 -c 4096 -n 512 -p "<a code-flavored prompt>"

MOE_TRACE_OUT=mymodel-chat.csv ./build/bin/llama-moe-trace -m model.gguf \
  -ngl 99 -ncmoe 99 -fa 1 -c 4096 -n 512 -p "<a chat-flavored prompt>"

cat mymodel-code.csv mymodel-chat.csv > mymodel-merged.csv

512 generated tokens per prompt is enough. Merge traces from contrasting workloads — a merged profile measures within 1% of per-workload specialist profiles, so one merged CSV per model is all you need.

2. Serve with the cache:

./build/bin/llama-server -m model.gguf -ngl 99 -ncmoe 99 -fa 1 \
  --moe-cache-profile mymodel-merged.csv --moe-cache-slots 112

Also works per model in a --models-preset INI section (moe-cache-profile = ..., moe-cache-slots = ...), and as env vars LLAMA_ARG_MOE_CACHE_PROFILE / LLAMA_ARG_MOE_CACHE_SLOTS (or legacy GGML_MOE_CACHE_PROFILE / GGML_MOE_CACHE_SLOTS, which llama-bench also accepts).

3. Confirm it engaged — look for this line at load:

init_moe_expert_cache: expert cache: 40 layers x 112 slots, 8164.00 MiB uploaded to CUDA0

A warning instead of this line means the cache fell back to baseline (see Tuning).

Tuning

• --moe-cache-slots is the main knob — experts cached per layer. Throughput rises with slot count until the pack no longer fits in VRAM. The pack is all-or-nothing: an oversized request logs pack allocation failed - expert cache disabled and runs at baseline speed (it does not partially fill). The warning reports the per-slot cost and the maximum count that could fit — set slots to that, minus headroom for KV/compute buffers which allocate afterwards.
• The cold and hot chains overlap by default. CPU graph splits run on a worker thread so the GPU hot chain executes concurrently with the CPU cold chain (--no-sched-async-cpu to disable; llama-bench --sched-async-cpu 0,1 benches both). Worth +4-5% with speculative decoding, ~±2% without it; outputs stay bit-identical either way.
• Leave ~900 MB of VRAM free beyond the pack. A slot count that loads can still crash on the first large prompt: runtime CUDA pool growth allocates beyond what the load-time check sees. Size slots against the biggest prompt you will serve, not against "it loaded".
• Fill VRAM to just under the ceiling, don't sweat the split. Near the maximum, a marginal MB is worth about the same as cache slots or as fully-resident layers (lower --n-cpu-moe). Pure -ncmoe 99 + max slots is the simple default; a hybrid (e.g. -ncmoe 30 + fewer slots) buys ~1% decode and ~3% prefill at best.
• Context size competes with the pack. KV grows with -c and shrinks the viable slot count. Compressing the KV cache (-ctk/-ctv, e.g. TurboQuant types) frees VRAM that converts directly into slots — often worth more than the KV precision costs.
• Speculative decoding stacks multiplicatively. --spec-type draft-mtp composes with the cache (+48% cache × +12% MTP ≈ +66% on Qwen); reserve ~1 GB for the draft context by dropping a few slots.
• Expert size decides the payoff. Big experts (GLM: ~5 MiB each) gain the most per slot; small experts need high slot counts before the win beats the dual-path overhead (~25% traffic coverage is roughly break-even). If VRAM only fits <15% of the expert count, expect single-digit gains (Laguna above).
• Profiles are model-specific, workload-tolerant. A wrong-workload profile still helps (+28% measured on Qwen worst-case) but loses about half the win; the merged profile recovers nearly all of it. Regenerate only if your usage changes character entirely.

Troubleshooting

Recent API changes

• Changelog for libllama API
• Changelog for llama-server REST API

Hot topics

• Hugging Face cache migration: models downloaded with -hf are now stored in the standard Hugging Face cache directory, enabling sharing with other HF tools.
• guide : using the new WebUI of llama.cpp
• guide : running gpt-oss with llama.cpp
• [FEEDBACK] Better packaging for llama.cpp to support downstream consumers 🤗
• Support for the gpt-oss model with native MXFP4 format has been added | PR | Collaboration with NVIDIA | Comment
• Multimodal support arrived in llama-server: #12898 | documentation
• VS Code extension for FIM completions: https://github.com/ggml-org/llama.vscode
• Vim/Neovim plugin for FIM completions: https://github.com/ggml-org/llama.vim
• Hugging Face Inference Endpoints now support GGUF out of the box! https://github.com/ggml-org/llama.cpp/discussions/9669
• Hugging Face GGUF editor: discussion | tool
• WebGPU support is now available in the browser, see a blog/demo introducing it here.

---

Quick start

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models
• Release process

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
• sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain