moe-miss-substitution

Code, run scripts and result records for the paper

Which Expert Misses Are Worth Waiting For? Substitution and Caching for Mixture-of-Experts Inference on a Consumer GPU ChaeWoo Son. Research Square preprint, 2026. https://doi.org/10.21203/rs.3.rs-11268552/v1

The paper studies what happens when a mixture-of-experts (MoE) model with hundreds of experts per layer (mainly Qwen3-Next-80B-A3B) runs on a 24 GB consumer GPU with only a quarter of its experts in GPU memory and the rest read from an NVMe SSD. Experts missing from GPU memory are either fetched synchronously or substituted with experts already in memory, and the paper asks which substitutions account for the quality loss and whether the substituted gate weight (mass) can guide cache policies.

This repository keeps the whole path of the project, including early explorations that the paper does not use, in the order in which the runs were made.

Layout

The path of the project

Runs are numbered R1-R21 in the paper. Inside the code, comments and log headers call them "Part n" or "round n" (Part n = Rn). Internal names: xc / xcache = the engine's expert cache; E = fetch every miss (reference); T = the 0.15 rule; C = the first-choice rule; F = frequency-based eviction; A = background admission (rent-or-buy); Q = T+F+A (quality-first); S = C+F+A (speed-first); H = C with a 12.5% cache.

The paper uses R4, R5, R7 and R12-R21; the other runs were earlier explorations. Results that the paper reports as exploratory, chosen from the results, or planned in advance are marked that way in the paper.

Running

• The experiments ran on rented cloud GPUs (RTX 3090 and A40 pods). Each run_*.sh is the script a pod ran: it downloads the model and data from Hugging Face, builds what it needs, runs, and prints its results into the log. Paths such as /workspace refer to the pod's volume.
• test_*.py are local checks on tiny random models (for example, that our layer-wise forward pass matches transformers). They write temporary models under /tmp/moe_test/.
• Python 3 with PyTorch and transformers (4.57.6 for R21); the emulation of Qwen3-Next reuses transformers modules.
• Dataset and model revisions were not pinned in most runs (see Appendix C of the paper); R20s and R21 record them.

Data and models

Sessions are built by prep_data.py from public datasets: Wikipedia (Korean and English, wikimedia/wikipedia 20231101), codeparrot/codeparrot-clean-valid, HuggingFaceH4/ultrachat_200k, maywell/koVast and AI-MO/NuminaMath-CoT. The generation test uses GSM8K and HumanEval. Models: Qwen3-Next-80B-A3B-Instruct (BF16, and the Q4_K_M GGUF by unsloth), DeepSeek-V4-Flash and gpt-oss-120b.

Notes

• The code and the run scripts were written and maintained with Claude (Anthropic), as described in the paper's section on the use of an AI system.
• Some comments refer to planning documents and research notes that are not part of this repository.
• License: MIT (see LICENSE). The engine patch applies to llama.cpp, which is also MIT-licensed.

Citation

@misc{son2026expertmisses,
  author       = {Son, ChaeWoo},
  title        = {Which Expert Misses Are Worth Waiting For? Substitution and Caching for Mixture-of-Experts Inference on a Consumer GPU},
  year         = {2026},
  howpublished = {Research Square preprint},
  doi          = {10.21203/rs.3.rs-11268552/v1},
  url          = {https://doi.org/10.21203/rs.3.rs-11268552/v1}
}