moe-offload-bench

Real, on-hardware measurements of MoE expert-weight offloading on a resource-constrained box: does prefetching mixture-of-experts weights off disk, ahead of the matmuls that need them, actually help — and if so, which mechanism does the work?

Model: OLMoE-1B-7B-0924 (GGUF, 64 experts/layer, top-8 routing, 16 layers), run through a custom llama.cpp example harness (harness/expert-log.cpp) that hooks ggml_backend_sched's eval callback to see expert routing decisions in real time and prefetch the selected experts' weight slabs before the matmuls that consume them run.

Everything here is a real process run on real hardware — no simulation, no synthetic timing model. All logs and CSVs backing every number below are committed under results/.

Hardware

This is not a datacenter box. It's the kind of machine MoE weight-offloading would need to target for it to matter: RAM well under the model size, a single-digit-core CPU, and a disk that is the actual bottleneck for most of this workload.

Measured hardware ceilings (both regenerated fresh for this write-up; see results/io_ceiling/ and results/compute_ceiling/):

• I/O ceiling: ~150 MB/s. O_DIRECT sequential dd reads off the root disk, three 1024/512 MiB samples at different offsets: 145–157 MB/s. This bypasses the page cache entirely, so it's the raw disk's ceiling, not anything our harness does.

• Compute ceiling: ~2.0 tok/s. A dense, fully-RAM-resident 630M-param model (Qwen2.5-0.5B-Instruct, Q8_0, 638.74 MiB) run twice back-to-back via llama-bench (-p 0 -n 32 -r 1 --no-warmup, 2 threads): 2.10 tok/s cold, 2.03 tok/s warm. Cold and warm being nearly identical (not a page-cache warm-up effect) confirms this is a genuine compute measurement, not one still partly gated on disk.

  An earlier, unlogged estimate mid-project had put this figure closer to ~1.0 tok/s; that number was never saved to a committed log and could not be reproduced when re-measured for this write-up, most likely because this VM's actual CPU allocation varies with host load (see Caveats). The ~2.0 tok/s figure is the one with a log backing it and is what the ratios below use.

Harness design

harness/expert-log.cpp is a llama.cpp example binary (llama-expert-log) built against upstream llama.cpp with a tiny patch (harness/llama.cpp.patch, +9 lines: one exported llama_model_get_tensor() accessor). It does four things:

1. Reads GGUF tensor metadata directly, independent of loading the model, via the low-level gguf.h API — tensor name, ne[] shape, byte size, file offset. Expert tensors are stored as merged 3D tensors ([n_embd, n_ff, n_expert] for gate/up, [n_ff, n_embd, n_expert] for down) with expert as the outermost dimension, so each expert's slab is a contiguous byte range — base_offset + expert_id * bytes_per_expert. This is what makes byte-accounting exact rather than estimated: bytes_per_token in every table below comes straight out of this metadata, and it matched the harness's own pread() totals exactly in every run.

2. Hooks ggml_backend_sched_eval_callback on tensors named "ffn_moe_topk-<layer>" — the top-k expert-selection output that llama.cpp's build_moe_ffn() already tags. This fires with the real routing decision, strictly before the gate/up/down matmul nodes for that layer execute, and is the only per-token, per-layer hook point GGML exposes (there is no per-expert hook — gate/up/down each run as one batched node over all n_used experts, not one node per expert).

3. Prefetches each routed expert's gate/up/down slabs: a buffered pread() (no O_DIRECT) into a reused scratch buffer, which warms the kernel page cache for that byte range, followed by madvise(MADV_WILLNEED) on the same range in the tensor's real mmap'd address. No userspace cache, no custom eviction policy — the kernel's own page cache is the only cache. This is the design that survived (see Finding 1 for why).

4. mlock()s every non-expert tensor (embeddings, attention, norms, output head — 297.2 MiB, 147 tensors) unconditionally at startup, so the kernel never evicts the always-needed working set and the full page-cache budget goes to expert slabs, the only thing re-fetched every token (Finding 7).

Per-token instrumentation: wall-clock decode time, getrusage().ru_majflt (major page faults — pages that required a real fetch from backing store, not served from cache) taken as a delta around each llama_decode() call, and pread()/madvise() timing buckets. /proc/self/io's read_bytes field was tried as an independent check on real disk bytes but reads back 0 in this container environment the whole project (an LXC/Crostini limitation, not a code bug) — ru_majflt was the working substitute.

The seven findings

Each finding names the exact config and points at the log/CSV in results/ it came from. Full tables are in RESULTS.md.

1. A custom userspace cache never beat the kernel's page cache — and wasn't memory-safe

First design: O_DIRECT reads + a userspace LRU slab cache + mprotect()-ing expert tensors writable so cache hits/misses could memcpy() straight into the mmap'd weight memory. Swept the LRU budget 0/256/512/800/1200 MB on Q4_K_M, 32 tokens (results/sweep/):

tok/s got worse as the cache budget (and hit rate) went up — LRU bookkeeping and O_DIRECT overhead outweighed the disk-read savings at every budget tried. At 1200 MB the process didn't just get slow, it died: RSS climbed 1091→1469→1619→1733 MiB over four tokens before a fifth took 2269 seconds and the box hit virtio_balloon: Out of puff (genuine host memory pressure). Root cause: every cache hit or miss wrote into the mmap'd weight memory, and on a MAP_PRIVATE mapping that write is copy-on-write — a permanent, unreclaimable anonymous page. With no swap on this box, that RSS growth had nowhere to go but OOM. This whole design was scrapped. The eventual replacement (pure pread() + madvise(WILLNEED), no userspace cache at all, Finding 7) reached 0.114 tok/s — 2.5x the best number this design ever produced, safely.

2. Expert routing has no small hot set to cache

64-token generation, all 16 layers (results/expert_skew.csv, via scripts/analyze_skew.py): covering 80% of a layer's activations takes 20–33 of its 64 experts (median ~25), and 44–59 distinct experts get touched per layer within just 64 tokens. Routing is close to uniform. There's no small "hot" subset of experts worth special-casing a cache around — any useful cache has to be able to hold most of the expert population, which on this hardware it can't.

3. Token-to-token expert locality is low

Same 64-token run (results/locality.csv, via scripts/analyze_locality.py): adjacent tokens (distance 1) share on average only 35.7% of their 8 routed experts per layer; at distance 8 that drops to 20.8%. A "cache what the last token used" policy would miss roughly two-thirds of the time even one token later. This and Finding 2 together are why no expert-frequency or recency cache was pursued further — the workload doesn't have the locality such a cache needs.

4. tok/s does not scale linearly with bytes/token

Q4_K_M vs Q2_K, identical prompt, 32 tokens, pread+madvise design (results/final/):

If tok/s were purely 1/bytes, Q2 should be 1.67x faster than Q4. It's actually 1.92x faster — Q4 pays a bigger-than-proportional penalty for its larger footprint. Finding 5 is why.

5. The extra penalty is real, disk-driven page-fault pressure the harness's own timer doesn't see

Same two runs, major faults per token:

9.3x more major faults for only 1.67x more bytes. The harness's synchronous pread() timer only measures its own reads; this confirms — rather than just implies — that Q4's larger per-token working set causes disproportionate page-cache eviction-and-refetch churn during compute itself, on top of what the harness explicitly prefetches. (/proc/self/io would have measured this directly but reads 0 in this container, per the harness design notes above — ru_majflt is the substitute, and it's unambiguous here.)

6. Even the fastest quant tested is far below the hardware's compute ceiling

Q2_K's 0.175–0.201 tok/s vs the ~2.0 tok/s dense-model compute ceiling (Hardware section above) is roughly a 10x gap. This workload is I/O-bound, not compute-bound, at every quant level tested — which is the whole reason prefetch/caching work on the I/O side (Findings 1, 7) is where the effort in this project went.

7. mlock() is the one prefetch optimization that won; two more were tried and reverted

All four configs below are Q4_K_M, 32 tokens, pread+madvise prefetch, same prompt:

• mlock (win): locking the 297.2 MiB non-expert working set (embeddings, attention, norms, output head) so it's never evicted leaves the full page-cache budget for expert slabs — the only thing re-fetched every token. +26% tok/s, majflt cut 2.9x. This is the only optimization attempt in this project that improved on the plain pread+madvise baseline.
• Write-in-place (reverted): eliminating the scratch-buffer copy by pread()-ing straight into the tensor's own mmap'd address required making that mapping MAP_SHARED+PROT_WRITE (it was read-only). Every prefetched page came back dirty and needed writeback to disk — that write traffic contended with read traffic for the same disk queue and made things over 2x slower despite a genuine ~10% drop in major faults.
• Background thread pool (reverted): issuing prefetch pread()s from a worker thread so compute for one layer could overlap disk reads for the next was meant to hide I/O behind compute. It did the opposite — one background pread() thread and the compute thread's own page faults contending for the same disk queue cost far more than any overlap saved, and majflt/token rose 7.4x. There's no idle core on a 2-core box to hide that contention behind, and this disk didn't show the concurrent-queue-depth throughput gain a network download earlier in the project had (6 parallel HTTP range requests: 1.3 → 23 MB/s) — local disk and remote CDN throttling are different bottlenecks.

Time breakdown at the best config (mlock + pread + madvise, Q4_K_M, results/step4/q4_32tok_buckets.log): of 8.767s/token, pread is 5.091s (58%), real compute is 3.658s (42%), and madvise + the harness's own CSV-logging bookkeeping are both noise (<0.2% combined, confirmed by re-running with logging fully disabled: 0.109 tok/s, statistically indistinguishable from 0.114 tok/s with logging on). pread, gated by real disk throughput, is the floor — and per Finding 7's other two rows, this box has no cheap way to shrink it further.

Reproducing this

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 9a286ac98d2cab74231bd3f1fc3f2b8bdf05422e   # commit this was built against
git apply /path/to/moe-offload-bench/harness/llama.cpp.patch
cp /path/to/moe-offload-bench/harness/expert-log.cpp examples/expert-log/
cp /path/to/moe-offload-bench/harness/CMakeLists.txt examples/expert-log/
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-expert-log -j2

./build/bin/llama-expert-log -m /path/to/olmoe-1b-7b-0924-q4_k_m.gguf \
    -o results.csv -p "The history of the Roman Empire begins with" -n 32

CLI flags: -q disables per-expert CSV logging (measures instrumentation cost, Finding 7). mlock of non-expert tensors is unconditional.

Caveats

• This is a single shared/variable VM, not an isolated benchmark rig — dmesg shows virtio_balloon: Out of puff events during the heaviest runs, and repeated measurements of the same config vary by 10-20% run to run (occasionally more; the compute-ceiling re-measurement for this write-up came back ~2x higher than an earlier unlogged estimate). Every number in this repo is real and reproducible in direction, but treat absolute figures as "this box, this hour," not a portable hardware spec.
• All findings use n=32 generated tokens per run (single run per config, not averaged across repeats) — enough to see clear, consistent, order-of-magnitude effects, not enough to quote a confidence interval on a ±5% difference.
• results/ also contains earlier one-off debugging runs (expert_activations_meta_test*, expert_activations_naive2*, validate_cache.csv, etc.) from before the harness reached its current form. They're kept for the record but aren't referenced above.