moe-cache

Run Mixture-of-Experts models that are bigger than your RAM, without the speed cliff.
A plugin for an unmodified llama-server. No fork, no patch.

Why

MoE models such as Qwen3, gpt-oss or DeepSeek only use a few "experts" per token, but stock llama.cpp keeps all expert weights in memory-mapped files. When they do not fit in RAM, the OS page cache thrashes and speed falls off a cliff.

moe-cache manages the CPU-side experts itself:

• keeps the hot experts in RAM and reads the rest from your SSD with large, parallel direct reads;
• sizes its cache from your real memory limit and stays under it (no out-of-memory kills);
• learns which experts you use and warm-starts from that next time (moe-cache-learn builds a profile in a minute);
• produces token-identical output to stock llama.cpp (checked on every model in RESULTS; the Hunyuan test was short only);
• works with speculative decoding (MTP; it helps when the model or most of it fits: Qwen3.6 at 12 GB 21.7 -> 24.0 tok/s, but it was 17 % slower on Qwen3.8-Flash-Next with an 8 GB card), split GGUF files, MXFP4 / K-quants / IQ4_XS / Q5_0, CPU-only machines and Vulkan GPUs;
• runs long prompts on the GPU like stock llama.cpp does;
• expert executor (on by default): while missing experts load from the SSD it already computes the ones in RAM, and on hybrid Intel CPUs it can use the efficiency cores too. Results are bit-identical (every output was compared byte for byte with llama.cpp's own on the test models);
• faster Q2_0 kernel: llama.cpp has no SIMD Q2_0 dot product on x86; the plugin supplies one (bit-identical, about 3x faster), which matters for models whose down projections are Q2_0 (for example Qwen3.8-Flash-Next).

When to use something else

Get started

You need: Linux on x86-64, a llama-server built with shared ggml libraries (tested with llama.cpp commits 23b0202a1 (ggml 0.26.0, backend API 3) and 7fe450e19 (ggml 0.25.1, API 2); each needs its own plugin build, see docs/QUICKSTART.md), and a MoE model as a .gguf file. A GPU is optional. Ollama user? The models in ollama list are plain GGUF files: the path is on the line starting with FROM in ollama show NAME --modelfile.

1. Install (pick one)

2. Check your setup (changes nothing)

bin/moe-cache-check /path/to/model.gguf --llama-server /path/to/llama-server

Every line says OK, WARN or FAIL and what to do about it.

3. Run your model

bin/moe-cache-server /path/to/model.gguf --ram 12 --llama-server /path/to/llama-server --open

--ram 12 is the memory (GB) it may use; leave it out to use what is free. Add --fast for about +10 % on models much bigger than your RAM, at the cost of output that is no longer identical to stock (the launcher suggests it when it would help). The launcher plans the GPU/CPU split from your free VRAM, starts llama-server with the plugin and opens the chat page (the same address, http://localhost:8080, is an OpenAI-compatible API for editors, Open WebUI and scripts).

4. Optional: prove it is correct and measure it on your machine

tests/identity.sh /path/to/model.gguf --llama-server /path/to/llama-server      # PASS = the same tokens as stock llama.cpp

More detail, troubleshooting and the Vulkan build: docs/QUICKSTART.md.

Results (measured, clean runs)

Re-measured on 2026-10-15 with the current code where the model is still on the test machine (RESULTS section 44); older rows say which version they come from.

Prompt processing (prefill)

For long prompts llama.cpp already computes the experts on the GPU, so the prefill gain is smaller than the decode gain; it grows with how far the model exceeds RAM. Bigger prompt chunks on a 12-16 GB card are the next step (BIGGER_MACHINES B13).

Conditions: "RAM" is a memory limit set with a cgroup on a 31 GB machine; before every limited stock run the model file is dropped from the page cache (otherwise stock can read it from cache that does not count against the limit and looks up to 4.6x faster than it is, RESULTS 44). One run per row unless the section says paired. When the model fits in the memory llama-server may use, stock llama.cpp is faster (its page cache holds the whole file and stays warm across restarts), so moe-cache-server starts stock llama-server then and says so.

Tip for Intel 12th gen and newer laptops (performance + efficiency cores): pin llama.cpp to one thread per performance core, e.g. -C 0x555 --cpu-strict 1 on a 6-performance-core CPU (this alone gave stock llama.cpp +15 % decode speed on our laptop; moe-cache-server does it for you). Details in docs/RESULTS.md, section 11.

Methods, all numbers and every caveat: docs/RESULTS.md. Hardware for all measurements: Intel i7-13620H, 31 GB RAM, RTX 4060 8 GB, NVMe SSD.

Hardware and OS support

Benchmark it on your own model and your own use case

The benchmark tool (web page, standard protocol, many-model matrix, quality harness) now lives in its own repository: moe-cache-benchmark. Clone it next to this folder (or set MOE_CACHE_DIR) and it finds the plugin and the planner by itself.

How it works

 llama.cpp builds the graph of the model
          |
  ggml scheduler hands each operation to a backend
          |
   +------+------------------+
   |  GPU backend            |  attention, norms, shared layers, the experts you put on the GPU
   |  CPU backend            |  everything else
   |  moe-cache backend  <-----+  the CPU-side expert matrix multiplications:
   +-------------------------+    1. look at which experts the router picked
                                  2. make sure those experts are in RAM (read the missing ones from the SSD, in the background)
                                  3. compute the experts that are ready while the others still load (several threads, optionally efficiency cores)
                                  4. give back the memory of experts nobody needs (LRU)
                                  5. use llama.cpp's own dot-product functions, so every number is identical

Each expert keeps its normal address inside one large reserved block of memory; moe-cache decides which experts have real memory behind them. It plugs into ggml, not into llama.cpp's model code, so a model that llama.cpp runs through its expert multiplication should work; the architectures that were actually tested are listed in COMPATIBILITY and RESULTS.

Known limitations

• Long prompts (32+ tokens per batch) run their expert blocks on the GPU, as stock llama.cpp does, so prompt processing is at stock speed when the model fits (DeepSeek-V2-Lite: 166 to 183 tok/s vs 166 to 188 stock) and much faster when memory is short.

• It does not manage the GPU; which layers' experts sit in VRAM is chosen once at start by the planner. A GPU expert tier was evaluated and is not built, see docs/ENGINE_DECISION.md.

• Not every feature works with every llama.cpp build. How the plugin is loaded depends on how llama.cpp was built (moe-cache-server picks it):

  Feature	LD_PRELOAD (llama.cpp that links its backends directly: distro packages, ./install.sh)	GGML_BACKEND_PATH (backends as separate files: official releases, Vulkan builds, Windows)
  RAM expert cache, SSD streaming, memory limit, warm start	yes	yes
  Expert executor (computes ready experts while others load)	yes	yes (since 0.2.3.3; tested on a Vulkan build)
  Fast Q2_0 / IQ3_XXS kernels	yes	inside the executor only
  Working together with llama.cpp's GPU expert cache (--gpu-cache)	yes	no

  The moe-cache: active: ... line in the server log says which parts are on.

• Needs a llama.cpp with shared ggml libraries; tested with commits 23b0202a1 (ggml 0.26.0) and 7fe450e19 (ggml 0.25.1). The plugin refuses to load into a ggml with a different backend API than it was built for (it prints why and the model loads normally). Linux only for now.

FAQ

Do I need this if my model fits in RAM? No. Use stock llama.cpp with --moe-cache-mib (see "When to use something else" above). The gain is when memory is short or the model is larger than your RAM. work in progress though, I'm currently seeing different ways to improve speed and efficiency

Why is it not faster when my model already fits in RAM? Because there is nothing to fix: stock llama.cpp then runs from the page cache with the same kernels. On Qwen3.6-35B at 24 GB, decode needs about 0.5 GB of expert weights per token (8 experts x 32 CPU layers x 1.9 MB), which the memory bus delivers at about 35 GB/s, close to this laptop's measured maximum (40 GB/s), so it is bandwidth-bound for everyone (stock 40, moe-cache 40 tok/s). The speed-ups appear when memory is short (the same model at 12 GB: 10 to 23 tok/s), for models larger than RAM, and for models with slow kernels (Q2_0 and the IQ types, where the executor and the faster kernel help).

Does it use my GPU? Yes, through llama.cpp as usual. moe-cache-server picks how many layers' experts live on the GPU from your free VRAM (that alone is worth about +60 % over keeping everything on the CPU). moe-cache itself manages the experts that stay on the CPU side.

Is there a GUI? llama-server already ships a web UI (moe-cache-server --open). For a fancier chat app, point Open WebUI at http://localhost:8080/v1 (OpenAI-compatible API; we have not tested that combination ourselves).

Why not an own inference engine like Strata or Maya? Short answer: they are single-model engines with a high hardware floor; moe-cache is a small layer for the MoE models llama.cpp runs (tested list in COMPATIBILITY). The reasoning and our measurements: docs/ENGINE_DECISION.md.

Documentation

Quick start | Benchmarks | Results | Compatibility | Roadmap | Engine decision | Security and privacy

Configuration (environment variables; moe-cache-server sets them for you)

Support

moe-cache is built and tested in my spare time on my own machines. If it saves you a RAM upgrade, you can support it on Ko-fi.

Acknowledgements

• llama.cpp and ggml (MIT, Copyright The ggml authors): moe-cache is a plugin for their backend interface, and it runs their CPU kernels unchanged.
• Related projects with a similar approach (tiering experts across VRAM, RAM and SSD), built as single-model engines: Strata (MIT, by Niko1221) and Project Maya (MIT, by mw00).

License

• moe-cache is MIT-licensed, see LICENSE.
• third_party/ggml/ contains interface headers copied from ggml / llama.cpp (MIT, Copyright The ggml authors). Their license text is kept in third_party/ggml/LICENSE.
• moe-cache does not include any model weights. Models you run with it keep their own licenses.
• Product and model names (Qwen, gpt-oss, DeepSeek, Granite, Hunyuan, KAT-Coder and others) belong to their owners and are used only to say what was tested.