Shadowgraph
TrendingBuzzTop stargazers
Chrome extension
Sign in

IbrahimSiddiqui007/moe-cache-benchmark

Benchmarks for moe-cache: stock llama.cpp vs the moe-cache plugin on your machine, model and prompts, with a token-identity check

Python

1

1 commits

updated Oct 14, 2026

See the code

See what people are saying

SourceMessageScoreDate

moe-cache: MoE models bigger than your RAM, 2-12x faster on stock llama.cpp with identical tokens (Qwen3.8-Flash-Next 68 GB @ 24 GB RAM: 5.2 → 12.5 tok/s, Qwen3.6-35B @ 12 GB RAM: 6.9 → 22.6 tok/s) (r/LocalLLM)

**TL;DR:** moe-cache is a plugin for unmodified llama-server (no fork, no patch). When a MoE model doesn't fit in RAM, stock llama.cpp falls off a speed cliff because the OS page cache thrashes. moe-cache manages the CPU-side experts itself, and in the default mode the output tokens are identical…

23

Oct 16, 2026

README

moe-cache-benchmark

Benchmarks for moe-cache: stock llama.cpp against llama.cpp with the moe-cache plugin, on your machine, with your model and prompts, under a memory limit you choose, with a check that both produce the same tokens. Split out of the moe-cache repository on 2026-10-14 (from moe-cache commit b59288f).

What you need

  • Linux, Python 3 (standard library only), systemd-run for the memory limits.
  • A llama-server built with shared ggml libraries, and a built moe-cache next to this folder:
git clone https://github.com/IbrahimSiddiqui007/moe-cache && (cd moe-cache && ./build.sh)
git clone https://github.com/IbrahimSiddiqui007/moe-cache-benchmark

The tool finds moe-cache (plugin build and planner) in this order: MOE_CACHE_DIR, a moe-cache folder next to this one, ~/moe-cache-setup/moe-cache (where moe-cache's install.sh puts it), ~/moe-cache.

Contents

PathWhat it is
bin/moe-cache-benchweb page for runs: speed, identity, your own prompts, deep metrics (http://127.0.0.1:8765)
bin/moe-cache-bench-watchterminal view of a running benchmark
bench/standard.pythe standard protocol STP-1 (docs/TEST_PROTOCOL.md)
bench/run_matrix.pymany models x placements x memory limits, publishable table
bench/quality.pyquality harness: compare, longctx, soak, judge-calibrate, codegen
bench/coding_session.pycoding-session benchmark (prompt reuse on a large context)
tools/wait_cool.shthe clean-run gate as one command
docs/BENCHMARKS.mdwhat is measured and how to run capability suites

Using it

The tool (bin/moe-cache-bench) needs only the Python standard library. It measures stock llama.cpp and moe-cache on your machine, with your model and your prompts, under a memory limit you choose, and checks that both produce the same tokens.

bin/moe-cache-bench            # starts the tool and opens http://127.0.0.1:8765
  1. Model: give the path of your .gguf (the llama-server and plugin paths are filled in when found).
  2. Memory limit (GB): the RAM you want to emulate for the model. A model larger than this limit is the case moe-cache is made for; set 2 or 3 limits (for example 12 and 24) to see the curve.
  3. Modes: tick stock and moe-cache; leave "both orders" on (it runs forward and reverse to cancel drift).
  4. Your own prompts (optional): paste prompts separated by a line with only ---; add EXPECT: word lines to mark words the answer must contain. They are run on both modes and compared.
  5. Press Start. Watch it in the browser, or in a terminal with bin/moe-cache-bench-watch.
  6. Read the result: steady generation speed (mean of requests 2 to 4), time to first token, SSD read per token, cache hit rate, and PASS/FAIL for identical tokens. Runs that started hot or with background load are flagged and should be repeated.

Command line, for a fixed comparable protocol (two placements, two limits, both orders; see docs/TEST_PROTOCOL.md):

bin/moe-cache-bench --no-browser &
bench/standard.py MODEL.gguf --limits 12,24 --profiles cpu,gpu

The shareable HTML report and JSON export contain no prompts, outputs or local paths; the unfiltered versions are for you only (see docs/SECURITY.md). Benchmark every model on your disk (all Ollama models and all GGUF files in a folder, both placements, several memory limits) and get a table to publish:

bench/run_matrix.py --ollama --dir ~/models           # prints the plan: models, cells, limits
bench/run_matrix.py --ollama --dir ~/models --run     # runs it; writes results-matrix-DATE.md and .csv

What the benchmarks mean and how to run the usual capability suites against the same server: docs/BENCHMARKS.md.

Status

Feature-frozen: bugs are fixed, new features are not planned. Only clean runs are quoted (the tool flags runs that started hot or under load).

License

MIT, see LICENSE.

Similar repos

IbrahimSiddiqui007/moe-cache

Run MoE models bigger than your RAM, without the speed cliff

15

same org

devbitsxyz/ollama-bench

9

eugr/llama-benchy

llama-benchy - llama-bench style benchmarking tool for all backends

760

runamu/sysproj_icl_cache

0

mem0ai/memory-benchmarks

Open-source evaluation suite for memory-augmented LLM systems

126

alexellis/rigmark

RigMark benchmarks local AI the way coding agents actually use it.

22

GGZ14/BetterBench

Real-world percentile benchmark for LLM inference servers: 1% A/B resolution, offline HTML reports.…

67

KushagraKanaujia/throttle

Find and prove inference savings on your own GPUs: $/M tokens with 95% CIs, agent workloads,…

7

IbrahimSiddiqui007/moe-cache-benchmark

Benchmarks for moe-cache: stock llama.cpp vs the moe-cache plugin on your machine, model and prompts, with a token-identity check

Python

1

1 commits

updated Oct 14, 2026

See the code

See what people are saying

SourceMessageScoreDate

moe-cache: MoE models bigger than your RAM, 2-12x faster on stock llama.cpp with identical tokens (Qwen3.8-Flash-Next 68 GB @ 24 GB RAM: 5.2 → 12.5 tok/s, Qwen3.6-35B @ 12 GB RAM: 6.9 → 22.6 tok/s) (r/LocalLLM)

**TL;DR:** moe-cache is a plugin for unmodified llama-server (no fork, no patch). When a MoE model doesn't fit in RAM, stock llama.cpp falls off a speed cliff because the OS page cache thrashes. moe-cache manages the CPU-side experts itself, and in the default mode the output tokens are identical…

23

Oct 16, 2026

README

moe-cache-benchmark

Benchmarks for moe-cache: stock llama.cpp against llama.cpp with the moe-cache plugin, on your machine, with your model and prompts, under a memory limit you choose, with a check that both produce the same tokens. Split out of the moe-cache repository on 2026-10-14 (from moe-cache commit b59288f).

What you need

  • Linux, Python 3 (standard library only), systemd-run for the memory limits.
  • A llama-server built with shared ggml libraries, and a built moe-cache next to this folder:
git clone https://github.com/IbrahimSiddiqui007/moe-cache && (cd moe-cache && ./build.sh)
git clone https://github.com/IbrahimSiddiqui007/moe-cache-benchmark

The tool finds moe-cache (plugin build and planner) in this order: MOE_CACHE_DIR, a moe-cache folder next to this one, ~/moe-cache-setup/moe-cache (where moe-cache's install.sh puts it), ~/moe-cache.

Contents

PathWhat it is
bin/moe-cache-benchweb page for runs: speed, identity, your own prompts, deep metrics (http://127.0.0.1:8765)
bin/moe-cache-bench-watchterminal view of a running benchmark
bench/standard.pythe standard protocol STP-1 (docs/TEST_PROTOCOL.md)
bench/run_matrix.pymany models x placements x memory limits, publishable table
bench/quality.pyquality harness: compare, longctx, soak, judge-calibrate, codegen
bench/coding_session.pycoding-session benchmark (prompt reuse on a large context)
tools/wait_cool.shthe clean-run gate as one command
docs/BENCHMARKS.mdwhat is measured and how to run capability suites

Using it

The tool (bin/moe-cache-bench) needs only the Python standard library. It measures stock llama.cpp and moe-cache on your machine, with your model and your prompts, under a memory limit you choose, and checks that both produce the same tokens.

bin/moe-cache-bench            # starts the tool and opens http://127.0.0.1:8765
  1. Model: give the path of your .gguf (the llama-server and plugin paths are filled in when found).
  2. Memory limit (GB): the RAM you want to emulate for the model. A model larger than this limit is the case moe-cache is made for; set 2 or 3 limits (for example 12 and 24) to see the curve.
  3. Modes: tick stock and moe-cache; leave "both orders" on (it runs forward and reverse to cancel drift).
  4. Your own prompts (optional): paste prompts separated by a line with only ---; add EXPECT: word lines to mark words the answer must contain. They are run on both modes and compared.
  5. Press Start. Watch it in the browser, or in a terminal with bin/moe-cache-bench-watch.
  6. Read the result: steady generation speed (mean of requests 2 to 4), time to first token, SSD read per token, cache hit rate, and PASS/FAIL for identical tokens. Runs that started hot or with background load are flagged and should be repeated.

Command line, for a fixed comparable protocol (two placements, two limits, both orders; see docs/TEST_PROTOCOL.md):

bin/moe-cache-bench --no-browser &
bench/standard.py MODEL.gguf --limits 12,24 --profiles cpu,gpu

The shareable HTML report and JSON export contain no prompts, outputs or local paths; the unfiltered versions are for you only (see docs/SECURITY.md). Benchmark every model on your disk (all Ollama models and all GGUF files in a folder, both placements, several memory limits) and get a table to publish:

bench/run_matrix.py --ollama --dir ~/models           # prints the plan: models, cells, limits
bench/run_matrix.py --ollama --dir ~/models --run     # runs it; writes results-matrix-DATE.md and .csv

What the benchmarks mean and how to run the usual capability suites against the same server: docs/BENCHMARKS.md.

Status

Feature-frozen: bugs are fixed, new features are not planned. Only clean runs are quoted (the tool flags runs that started hot or under load).

License

MIT, see LICENSE.

Similar repos

IbrahimSiddiqui007/moe-cache

Run MoE models bigger than your RAM, without the speed cliff

15

same org

devbitsxyz/ollama-bench

9

eugr/llama-benchy

llama-benchy - llama-bench style benchmarking tool for all backends

760

runamu/sysproj_icl_cache

0

mem0ai/memory-benchmarks

Open-source evaluation suite for memory-augmented LLM systems

126

alexellis/rigmark

RigMark benchmarks local AI the way coding agents actually use it.

22

GGZ14/BetterBench

Real-world percentile benchmark for LLM inference servers: 1% A/B resolution, offline HTML reports.…

67

KushagraKanaujia/throttle

Find and prove inference savings on your own GPUs: $/M tokens with 95% CIs, agent workloads,…

7