moe-microscope

Building mixture-of-experts models small enough to understand.

MicroMoE is an educational and experimental mixture-of-experts model written in sw-MLPL. The repository isolates the techniques behind modern efficient MoE systems (routing, sparse dispatch, low-rank experts, recurrence, Engram memory, distillation, quantization, expert caching, heterogeneous execution) so each can be inspected and measured on its own, at a scale where every tensor, expert, routing decision, and byte can be printed. It is progressively constructing a tiny model inspired by FreeToken, HRM/TRM, DeepSeek's Engram, and sparse routing; it does not claim that the combined architecture is validated yet.

Large MoE systems make it hard to see what each mechanism buys. Here the whole model is tens of thousands of parameters, every lesson is a readable MLPL program that records its own intermediate values, every diagram is drawn from values a test asserts, and every claim in this page cites a results row or a measured document.

• Documentation landing page (three reader journeys)
• Architecture and delivery plan
• Results dashboard and the full results table

Live demo

The moving parts step by step, live on GitHub Pages: https://sw-ml-study.github.io/moe-microscope/. Pick a lesson (the dense baseline, the top-1 and top-2 mixtures, the recurrent block, the recurrent mixture, the Engram table, the three sparsities together, or one window through routing and dispatch) and step through its frames. The landscape page beside it (landscape.html) pans along the same recordings as moving data, scene by scene. Both are playbacks: no sw-MLPL code runs in the browser, and every number is read from a pinned fixture that a real training run or its export script wrote. To produce those fixtures yourself, run the training at the command line: just dense write, just moe write, just moe2 write, just recur write, just rm write, just engram write, just re write, just recordings write, and just walkthrough write (see Build and run below); just serve-learn then serves the same page locally. The blog post about this work: https://blog.softwarewrighter.com/2026/09/13/saw-building-a-tiny-mixture-of-experts/. The wiki mirrors the documentation as a navigation layer.

Status

Demonstrated, with measurements: a dense baseline (DN01); top-1 and top-2 mixtures (MX01, MX02); measurable expert specialization; exact sparse dispatch (SD01); the data-versus-epochs effect (DS01); low-rank delta experts with a shared expert (LD01); an in-repo teacher fixture (TE01); the resource budget (RB01) and a measured generation benchmark (GB01); recurrence (RC01, RM01); the Engram table from scratch with exact parity against the sw-MLPL builtin (EG01); all three sparsities together and the seven-row ablation matrix (RE01); seven recordings replayed in the live demo and handed to the generic Rust/Yew host; the campus docent's first five steps (paused: it does not yet beat a deterministic matcher).

Still to come, in order: the animated data-flow landscape (in progress), a Mamba-style state-space block and the attention, state-space, and MoE hybrid with latent experts, distillation, the state-memory-experts composition, multi-token prediction, the routed docent, quantization and the packed file with the expert cache, CPU/NPU scheduling and the 256 MB target, the interactive host, the configuration frontier, the CUDA move with host-resident experts, and the findings report. See the saga queue.

What we have learned

Every experiment is also graded and ranked against what it was expected to show in the report card.

1. Sparse capacity is real. The four-expert mixture stores 7,096 parameters and touches 3,064 per token; the dense model stores 3,812 and touches 2,996. Stored capacity grew 1.86 times for 1.02 times the active parameters, at the same validation loss (3.76 against 3.79). Evidence: MX01 and DN01 rows in the results table.

2. Experts specialize, without being told the task. Under top-1 routing the specialization score is 0.61 against a 0.25 family-blind baseline: prose leans on one expert, arithmetic on another. The router never saw a family tag. Scope: the four synthetic families are deliberately separable. Evidence: MX02.

3. Top-k is a cost and quality knob, not a free improvement. Top-2 doubles expert evaluations per token, fits the training rows better (0.86 against 0.62 exact match), gives the first held-out MLPL answers, and has a worse validation loss (3.92) and a flatter specialization map (0.42). Evidence: MX02 row.

4. Data moves the held-out columns; epochs do not. From 120 to 960 examples, dense validation loss falls 3.48 to 1.11 and the mixture 3.49 to 1.22; prose becomes a solved family held-out. Doubling epochs at 120 examples makes both worse. The mixture beats dense at 480 examples and loses at 960 at twice the cost. Evidence: DS01.

5. Sparse compute is not automatically faster. Sparse dispatch cuts expert row evaluations from 13,440 to 3,360 with outputs that agree exactly, and is slower in this interpreter (0.74 against 0.42 ms per window) because gather, scatter, and loop overhead exceed the three tiny matmuls skipped. Evidence: SD01 and the generation benchmark.

6. Cheap experts are worth trying, with a shared expert. Sixteen rank-4 delta experts cost 2,048 parameters against 4,288 for four full experts and give validation loss 3.46 with a shared always-on expert (the best non-recurrent row) and 3.79 without (the dense baseline's). Evidence: LD01.

7. Reusing a block buys quality per parameter, and the router re-decides. The mixture block applied three times with deep supervision reaches validation loss 3.42 at the same 7,096 parameters (MX01 3.76), and 38 to 46 percent of tokens change expert between recurrences. It is not a converging reasoning loop: the models score best when stopped early, and R=2 is worse than R=1. Evidence: RC01, RM01.

8. External memory is memorization capacity at this data size. The Engram table, written from scratch and equal to the sw-MLPL builtin to the bit in outputs and gradients, lowers training loss and raises validation loss at every table size (3.93 to 4.67 against 3.79); composed with routing and recurrence it reaches training loss 0.006 and the worst held-out row of the ablation (4.76). Evidence: EG01, RE01.

9. A trained docent does not yet beat keyword matching. On held-out paraphrases the deterministic matcher scores 0.685 and the best docent 0.407, so the docent stays out of the campus site until it earns its place. Evidence: CD01b.

How small is MicroMoE?

Every number is labeled measured, derived, or estimate in resource economics; the calculator behind it is pinned to the lessons' measured parameter counts.

One full expert is 1,072 parameters: 8,576 bytes at f64 and 536 bytes of INT4 payload; a rank-4 delta expert is 128 parameters. Generation runs at roughly 6,000 tokens per second dense, 3,200 top-1, 2,000 top-2, and 3,100 packed deltas in the interpreter on one laptop, with p95 token latency under 0.6 ms in every case (benchmark).

Architecture progression

Dense model, then routed experts, then top-k, then true sparse dispatch, then recurrence, then Engram memory, then quantization, then a bounded expert cache, then heterogeneous execution. Four independent sparsities organize it: parameter sharing (recurrence), conditional compute (MoE), conditional memory (Engram), and residency (cache); the state-space block (sequence state), quantization (precision), and multi-token prediction (time) extend the taxonomy to seven resource allocation axes. The architecture page has the full picture; the concept pages walk it one mechanism at a time.

Experiments

Each experiment introduces one mechanism and answers one question: experiment index. The machine-readable inventory is catalog/lessons.toml.

Next

Recurrence is measured (RC01: one block applied several times with deep supervision; RM01: routing over reasoning time, with the lowest 120-example validation loss so far at R=3), and so is the Engram lookup table from scratch (EG01: exact parity with the sw-MLPL builtin; at 90 training windows the table adds memorization, not generalization), and the three sparsities together (RE01: the seven-row ablation matrix, in which the composed model memorizes best and generalizes worst). The recurrence-and-Engram saga is closed. The campus docent (a tiny model trained in batch here to direct visitors of the research campus site) is paused after its first five steps: it does not yet beat a deterministic matcher. Next is an animated data-flow landscape (every mechanism's data movement played back from the pinned recordings, panning along one wide scene), then the routed docent, distillation, quantization, the expert cache, and heterogeneous execution. Plan: docs/overview/plan.md.

What is here

docs/README.md               Documentation landing page with three reader journeys
docs/overview/               Architecture, delivery plan, saga queue
docs/results/                Resource economics (RB01) and the results dashboard
docs/experiments/            One page per experiment
docs/reference/              Full results table, generation benchmark, capability ledger, upstream findings
docs/implementation/         Host handoff, sibling work orders, teacher fixture schema
docs/research/               The design discussion, the Saga 2 review, the MoE/Engram questions
lib/  demos/  tests/         MLPL model code, lessons, and native mlplunit tests
probes/                      Standalone reproducers re-checked by the gate
fixtures/  assets/previews/  Bounded fixtures, pinned recordings, and committed diagrams
catalog/                     Machine-readable lesson inventory
scripts/  justfile           Thin gate and tool-selection scripts

Build and run

Prerequisites:

• the adjacent ../sw-mlpl checkout built in release mode (target/release/mlpl-repl), or an absolute MLPL override;
• mlplunit on PATH, an absolute MLPLUNIT override, or the adjacent ../../softwarewrighter/mlplunit/bin/mlplunit checkout;
• just;
• agentrail for the development process.

The scripts only select existing tools; they never install or overwrite them.

just                 # list recipes
just check           # the complete pre-commit gate
just tests           # native mlplunit tests (arguments filter paths or tags)
just probes          # re-run the upstream-finding reproducers
just domain          # run DM01 and check its diagrams
just dense           # run DN01 (about 16 s) and check its diagrams and results row
just teacher         # validate the committed TE01 fixture without retraining
just router          # run the MX01 routing microscope and check its diagram
just moe             # run MX01 training (about 36 s) and check its diagrams and results row
just dispatch        # run SD01 sparse dispatch with exact parity (about 5 s)
just moe2            # run MX02 top-2 routing and the specialization map (about 36 s)
just scale           # validate the DS01 data-scale points and diagram (opt-in sweep: just scale write)
just delta           # run LD01 low-rank delta experts with a shared FFN (about 80 s)
just budget          # RB01 resource budget: document and diagram, no training
just benchmark       # GB01 generation benchmark fixture and document (write to re-measure)
just mlpl-style      # canonical formatting and docstring checks

just check validates repository structure, documentation links, peer-identical license files, the generated Agentrail briefing, MLPL style, the native test suites, the pinned reproducers, and the DM01 and DN01 lessons with their diagrams and results row (about half a minute in total). Each lesson extends the gate with its own demo run, preview freshness check, recording check, and catalog check.

The interactive host is the generic Rust/Yew/WASM microscope and MLPL web framework in ../demo-extensions; that Rust code is the only part of this effort subject to sw-checklist.

Development process

Work is divided into durable Agentrail steps. In each fresh session run agentrail next, then agentrail begin; implement only that step; run focused tests and just check; commit source and .agentrail/ metadata by name; push main; and only then run agentrail complete. AGENTS.md holds the full repository protocol and CLAUDE.md links to it.

Copyright and license

Copyright (c) 2026 Michael A Wright. See COPYRIGHT.

Distributed under the MIT License.