MoEspresso

Run large Mixture-of-Experts language models well on the memory you have.

MoEspresso is an inference engine for a deliberately small set of large MoE models on Apple Silicon. A package says exactly what it contains and how it must run, and the runtime builds the model from those declared facts. The priority is model quality, explicit contracts, and measured behavior.

What makes MoEspresso different

• Routed experts keep their own quantization format inside MLX. The DeepSeek-V4-Flash package stores its routed experts in the IQ_K formats and decodes them through mlx-iqk; its dense side is q6_K through mlx-kquant, which also carries the K-quant routed experts of the Ornith package. A TurboQuant path is implemented as well, though no published package uses it. None of this is limited to the affine-only MLX weight path.
• SSD expert streaming is an execution mode of the pooled routed runtime. Routed expert rows are stored contiguously for one-read fetches, and one expert pool covers both fully resident and SSD-backed execution. Streaming covers the K-quant, TurboQuant, and IQ_K routed formats.
• Defaults favor quality at long context. A package fixes a quality-gated quantization recipe. Serving uses a broadly useful 128K context window when the runtime can reserve it. On lower-memory DeepSeek-V4 hosts it reports a smaller safe default. The pooled runtime streams routed experts instead of changing weight precision or silently reducing model quality.
• Speculative decoding ships and engages on its own. A package can declare a bundled drafter. The DeepSeek-V4-Flash package carries one, and serving enables it when the memory budget covers it. See speculative decoding.

In numbers

DeepSeek-V4-Flash, full-resident on an M3 Max with 40 GPU cores and 128 GB unified memory. Lower NLL and perplexity are better; higher first-token agreement and decode throughput are better:

MoEspresso was not calibrated on this WikiText test panel. See accuracy, performance, and the comparison protocol.

Supported models

The DeepSeek-V4-Flash package is a whole-model 2.37 bits per weight over the served model's 284.335e9 parameters, with the drafter excluded from both sides of that ratio. Its routed experts average 2.2491 bits per weight as a mix of IQ2_KS on 89 of the 129 routed-expert tensors and IQ2_K on the other 40, and they hold 77.88 GB of the model shards. The remaining 6.47 GB is the dense side, stored in q6_K.

Install

MoEspresso 2.1.2 requires an arm64 Apple Silicon Mac running macOS 26.2 (Tahoe) or later.

Install MoEspresso with Homebrew. The formula installs its required Python runtime and native dependencies:

brew install steadfastgaze/tap/moespresso

Check the installed release and list its user-facing commands:

moespresso --version
moespresso --help

Quick start

Install the Hugging Face CLI with Homebrew:

brew install hf

Download the Ornith package into an explicit directory:

hf download steadfastgaze/Ornith-1.0-35B-Q4_K_M-MoEspresso \
  --local-dir ./models/ornith-35b

Or download the DeepSeek-V4-Flash package:

hf download steadfastgaze/DeepSeek-V4-Flash-0731-2.37bpw-MoEspressoV2 \
  --local-dir ./models/deepseek-v4-flash

You can verify the package:

moespresso verify ./models/ornith-35b

Verification checks the manifest's identity and validity, every declared member's path, size, and SHA-256, the tensor keys in every shard, and the manifest-derived sidecars. Start the local server:

moespresso serve ./models/ornith-35b --thinking off

The server performs one short isolated warmup before announcing readiness, then exposes POST /v1/chat/completions and GET /health on 127.0.0.1:8080 by default.

Leave that terminal running. With OpenCode installed, configure and start it from a second terminal:

moecode() {
  OPENCODE_CONFIG_CONTENT='{
    "provider": {
      "moespresso": {
        "npm": "@ai-sdk/openai-compatible",
        "name": "MoEspresso (local)",
        "options": { "baseURL": "http://127.0.0.1:8080/v1" },
        "models": {
          "MoEspresso": {
            "name": "model @ MoEspresso",
            "temperature": true,
            "interleaved": "reasoning_content",
            "limit": { "context": 131072, "output": 32768 }
          }
        }
      }
    },
    "model": "moespresso/MoEspresso",
    "small_model": "moespresso/MoEspresso"
  }' opencode "$@"
}

moecode

The model declaration lets OpenCode forward an agent temperature and preserve assistant reasoning in reasoning_content across turns. Thinking remains a server-startup choice through moespresso serve --thinking; the declaration does not advertise per-request reasoning-effort variants.

The main serving controls are:

• --host and --port choose the listen address.
• --thinking off|on|high|max selects the model's own thinking mode. high is an alias of on for every family. Ornith takes on or off through its packaged template. DeepSeek-V4-Flash maps the flag onto its official encoder modes: off renders chat mode (the default), on renders thinking mode, and max adds the official maximum reasoning-effort preamble; max refuses loudly for families without an effort mechanism. The selection is fixed at startup; per-request render fields stay rejected so the served prefix contract is stable.
• --prompt-cache-size and --prompt-cache-bytes bound in-memory prefix-cache retention.
• --startup-warmup off deliberately restores cold first-request behavior, mainly for measurement.
• --max-memory-gb sets the expert-pool capacity planner's startup ceiling. Its exact meaning matters, so it is described below.
• --max-context-tokens selects any positive context limit up to the package's architecture limit. The default is 128K or the package limit, whichever is smaller. DeepSeek-V4 may report a lower automatic default when the exact cache reservation would otherwise leave no usable expert pool.
• --min-resident-experts sets a floor on the pooled routed runtime's per-layer expert capacity and fails at startup when the loaded capacity is smaller. A package whose runtime reports no expert pool refuses to start under this flag.

Memory policy and expert residency

MoEspresso keeps attention, norms, routers, shared experts, and other non-routed weights resident. What happens to the routed side depends on the package's routed format.

The pooled routed runtime carries the K-quant, TurboQuant, and IQ_K routed formats. The DeepSeek-V4-Flash package uses it for IQ_K experts, while the Ornith package and the earlier DeepSeek K-quant packages use it for K-quant experts. At startup it reserves a fixed allowance for KV and activations, then spends the remaining planned budget on routed-expert slots. If all 256 experts fit, the default startup policy loads every row and full-capacity execution is the zero-miss, zero-on-demand-I/O case of the same pooled graph. If they do not, missing experts are read into persistent slots as routing selects them.

To run a pooled package without on-demand expert reads during inference, require the startup report to show full expert capacity (capacity=256). That is selected automatically when the budget admits every expert row; it is not a separate, lower- or higher-quality model graph.

For a fail-closed all-resident launch, request an explicit full prewarm:

MOESPRESSO_SSD_PREWARM_EXPERTS=all \
moespresso serve ./models/ornith-35b --thinking off

This loads every expert before serving and fails when the planned pool cannot hold all 256 experts. It exercises the same pooled routed graph as SSD streaming; the only difference is that every row is already resident.

Serving targets a 128K context limit. Each package retains its larger architecture limit, which can be selected explicitly with --max-context-tokens. For DeepSeek-V4, the default planner prices the cache from the package's attention geometry and the served context. When 128K cannot coexist with the minimum expert pool, it chooses the largest safe 1K-aligned default and prints auto_reduced_from=131072. An explicit context limit is not reduced; startup fails with a memory breakdown if it cannot fit. Any resolved limit below 128K prints a prominent warning because usability is substantially reduced for long conversations and agentic work.

MOESPRESSO_SSD_KV_ALLOWANCE_GB remains the explicit fixed-allowance override. Other model families use that fixed allowance because they do not yet publish a family-specific cache-size contract.

--max-memory-gb caps the input to that startup capacity calculation. It is not an RSS limit. It selects a smaller or larger base expert-pool capacity after subtracting the resident base, the resolved KV/activation allowance, and a safety margin. Serving can grow selected layers after a completed request when the adaptive-growth and replacement-memory budgets allow it. Pools never shrink as context grows, so operators must choose the ceiling and allowance for the context workload they intend to serve. MOESPRESSO_SSD_KV_ALLOWANCE_GB sets a fixed allowance (default: 1 GiB where the family has no exact contract). A capacity-capped run on a larger Mac reproduces pool geometry and hit behavior, but its SSD miss cost can be optimistic because macOS may retain the whole package in page cache.

See SSD streaming for the capacity formula, runtime controls, direct-read path, and measurement caveats.

Speculative decoding

A DeepSeek-V4-Flash package can declare a draft model as an optional bundled component. The published package carries DeepSeek's DSpark drafter, and serving enables it when the package is fully resident, every declared component file is present, and the wired-memory budget covers the weights, the drafter, cache state at the served context limit, and the per-request working set. When one of those does not hold, serving stays plain and prints the reason; automatic selection never refuses startup. The loop is lossless by rule: a proposed token is kept only when the target model's own verification accepts it, so no unverified token is ever emitted.

MOESPRESSO_DS4_DRAFTER=off moespresso serve ./models/deepseek-v4-flash  # no drafting

Use a separately downloaded DSpark sidecar by passing its directory to both verification and serving:

moespresso verify ./models/deepseek-v4-flash --drafter ./models/dspark
moespresso serve ./models/deepseek-v4-flash --drafter ./models/dspark

The command recognizes the sidecar family from the manifest at the supplied root. This release accepts DSpark on the external command surface and refuses DFlash or MTP sidecars with a clear error. --drafter overrides MOESPRESSO_DS4_DRAFTER and bundled automatic selection. The source tree retains the DFlash and MTP implementations for future model packages.

DSpark requests keep the in-memory prefix cache and the disk checkpoint tier described below, so a follow-up turn over a shared prefix resumes speculation instead of prefilling that prefix again. When the stored drafter state cannot be reused, the validated prompt cache still serves the request plainly. DFlash serves from a fresh per-request cache.

See speculative decoding for the drafter protocol, the families and their sidecars, the adaptive scheduler, the memory each sidecar costs, and the correctness contract.

Disk KV for restart-warm and cross-session resume

The in-memory prefix cache is always on. The disk KV tier checkpoints aligned prompt-cache frontiers so a later process, or a new session sharing a long prompt prefix (an agent client's fixed system prompt and tools), restores an exact token prefix and prefills only the suffix. Serving enables it by default under ~/.cache/moespresso/disk_kv/<package> with an 8 GiB LRU byte budget per package, a 1024-token stride, and a 16k-token write-depth cap (checkpoints cover the shared-prefix region; deep conversation tails are not snapshotted):

moespresso serve ./models/ornith-35b --thinking off
MOESPRESSO_DISK_KV=off moespresso serve ./models/ornith-35b   # memory-only

MOESPRESSO_DISK_KV_ROOT, MOESPRESSO_DISK_KV_STRIDE, MOESPRESSO_DISK_KV_BYTES (unlimited disables eviction), and MOESPRESSO_DISK_KV_WRITE_DEPTH (unlimited snapshots any depth) override the defaults. A root has one process owner, restores are package/render/KV-policy scoped, and a prompt-cache checkpoint that does not match fails closed to cold prefill. Deleting ~/.cache/moespresso is always safe. See the disk KV contract for the full guarantees.

Why a MoEspresso package exists

A package is more than a collection of quantized weights:

1. Its manifest declares architecture, tensor formats, required backend operations, tokenizer/rendering identity, content-addressed file identities, and a provenance chain through the plan and producer that created it.
2. Each routed layer stores one uint8 [n_experts, row_bytes] bundle. Row e is expert e's complete gate, up, and down payload laid out contiguously, so one missing expert can be fetched with one contiguous read instead of a scatter of projection reads.
3. The bundles live per layer inside ordinary safetensors shards; there is no single global expert file. Shard metadata carries the exact offsets, shapes, dtypes, codecs, and component order needed to index each row without reading weight data.

That layout is an engine/package co-design: the runtime knows exactly how to keep the all-resident case fast and, on the pooled routed formats, how to turn the same rows into SSD-backed expert slots. A package meant only as a generic weight container would not provide that contract. Direct GGUF loading or a more interchangeable package form may be supported in the future, but neither is the current runtime input.

The full contract is in package format.

Accuracy focus

Package, cache, attention, routing, and quantized-math changes are promoted through model-specific token or logit comparisons and must exercise the intended runtime path. DeepSeek Q0/Q1 fixtures are public material carried from the DwarfStar suite. Provider-derived Q2 continuations and top-logprob captures remain private by design.

DeepSeek-V4-Flash release gates

The DeepSeek-V4-Flash results below were measured on the published artifact against the DeepSeek-V4-Flash 0731 release, at full residency with the drafter off:

Two reference points frame those numbers. The official API's own pass-to-pass step-level agreement on the same 100 prompts is 89.67 percent, and the package sits about 2.3 points below that reproducibility reading. The bf16 teacher's perplexity on the same 32 WikiText windows is 4.8886.

Ornith

Ornith has a separate nine-item served gate spanning reasoning, agentic coding, and long-context recall.

The public Ornith NLL matrix complements that served gate with the same 6,132 teacher-forced target tokens and all 248,320 output logits in every arm:

The MoEspresso and mlx-lm teacher-forced measurements do not use a generation KV cache. The llama.cpp scorers use a fresh F16 KV cache for each window.

Further guarantees and limitations are documented in DeepSeek quality, Ornith quality, and the shared correctness ladder.

Performance focus

The DeepSeek-V4-Flash rows were measured on the published artifact, on a 128 GB unified-memory configuration: ten alternating fresh-process arms at a 3,844-token prompt, greedy decoding, the disk KV tier off (MOESPRESSO_DISK_KV=off), thermal gating before and after every arm, an 8-token warmup request ahead of each measured request, and every arm exact against the package's 38-token reference rail. The target used the pooled runtime at capacity 256 with all 129 projection pools identity-mapped and no request-time expert I/O.

Every drafter-on arm read above every drafter-off arm, and the smallest gap between the two sets is 4.92 tok/s. The median gain is 22.70 percent.

The 37,000-token product-stack comparison is summarized in numbers. Exact artifacts, engine revisions, prompt hashes, cache controls, thermal readings, and per-run results are in DeepSeek speed.

The Ornith comparison matrix was measured on an M3 Max with 40 GPU cores and 128 GB unified memory. Every cell is the median of three fresh-process runs with an 8-token prewarm and 256 measured output tokens. Engines were left-rotated between rounds; AC power, normal thermal state, greedy decoding, temperature 0, vision off, and speculation off were enforced. Runs used the memory-only cache path: the disk KV tier, on by default when serving, was disabled (MOESPRESSO_DISK_KV=off). Values are decode throughput / TTFT-derived prompt throughput, in tokens per second.

MoEspresso and mlx-lm use affine Q8 with group size 64. llama.cpp uses q8_0 K and V with group size 32. MoEspresso and llama.cpp use Q4_K_M weights from the same GGUF lineage. mlx-lm uses Jundot oQ4e weights. The MoEspresso column was produced by engine 1.0.0; the compared engines are mlx-lm 0.31.3 and llama.cpp at commit 6eddde0.

A separate matched mlx-lm diagnostic at 8,191 tokens measured 69.96 tok/s with raw BF16 KV and 63.33 tok/s with affine Q8 KV. Raw BF16 uses substantially more attention-cache storage, so Q8 remains the product comparison.

Read DeepSeek speed, Ornith speed, and the exact acquisition, prewarm, thermal, timing, repeat, and evidence protocol in benchmark reproduction.

Documentation

• Documentation map: model, package, runtime, quality, and benchmark references.
• Developer guide: lifecycle, source map, entry points, and test commands.
• Contributor guide: working rules and invariants.
• DeepSeek package recipe and Ornith package guide: developer-facing package construction, including Ornith's remaining public source-adapter gap.
• Resident runtime, SSD streaming, and package format: the core implementation contracts.
• Speculative decoding: drafter families and sidecars, the draft/verify loop, the scheduler, cache reuse, and memory.

Acknowledgements

MoEspresso would not exist without the ideas, code, measurements, and examples of a large community. This list cannot be exhaustive, but these debts are concrete:

• Iwan Kawrakow wrote the quantization formats MoEspresso serves. The DeepSeek-V4-Flash package stores its routed experts in his IQ_K formats, and the bytes are produced by his quantization algorithms, published in ik_llama.cpp and vendored at a pinned commit with their upstream notices in mlx-iqk, which the release installs from PyPI at 0.1.2 and which carries the Metal decode kernels and the k-contiguous relayout the routed path reads. The Ornith package and the earlier DeepSeek packages ship the k-quants, which he designed and implemented in llama.cpp. MoEspresso's Metal serving of the IQ_K formats is this project's own implementation, and its correctness target is his: the served bytes reconstruct bit for bit what his CPU dequantizers produce.
• Georgi Gerganov's ggml and llama.cpp built the ground this work stands on: the GGUF container MoEspresso's recipe importers read, the block-quantization ecosystem the served formats belong to, and a reference engine that anchors the published speed and perplexity comparisons.
• Asher Feldman's mlx-kquant made K-quant wire formats practical inside MLX. Finding that work was a turning point: after the model graph had been checked seam by seam but low-bit native weights still failed behavior gates, this extension raised the quantization quality on MLX. The release installs 0.3.0 from the fork at steadfastgaze/mlx-kquant, revision 6fbfd4f5, which carries the kernel work this project's serving paths depend on; the revision behind the published Ornith rows is recorded in benchmark reproduction.
• Bartowski's calibration_datav5 corpus (pinned gist 82ae9b520227f57d79ba04add13d0d0d, raw commit 14543ac) is the training spine of the DeepSeek-V4-Flash calibration set; his gist in turn credits Dampf, Kalomaze, and edaddario's datasets. His published GGUF quantizations also serve as external baselines in the benchmark documentation.
• turboderp's exllamav3 (pinned at commit 0b9745c) contributes the technical.utf8 calibration file and the held-out c4.utf8 and code.utf8 probes.
• DeepSeek trained and released DeepSeek-V4-Flash and its DSpark drafter checkpoint under the MIT license. The package repackages those weights, and the prompt renderer adapts the upstream encoding module, as recorded in THIRD-PARTY-NOTICES.
• froggeric's Qwen-Fixed-Chat-Templates (Apache-2.0, pinned at v19) provides the vendored Ornith chat template; MoEspresso's modifications are described in THIRD-PARTY-NOTICES.
• Jinho Jang's JANG (Apache-2.0) was a rich source of ideas, especially in its willingness to explore new quantization schemes. MoEspresso supports a distinct manifest-driven TurboQuant path built with JANG's codec components, and JANG remains part of the live DeepSeek-V4-Flash path through its MLX model graph and cache primitives. Several MoEspresso kernels adapt JANG kernel sources, as recorded in THIRD-PARTY-NOTICES.
• antirez's DwarfStar was more than a reference engine. MoEspresso began independently, but adding DeepSeek-V4-Flash against a serious, narrow, quality-and-speed-focused baseline gave the project a product signal and a standard worth refining toward.
• Jundot's oMLX brings continuous batching, tiered KV caching, and oQ mixed-precision work. It provided an important performance reference and may influence this project further.
• Apple and the MLX and mlx-lm communities provided the array framework, unified-memory model, graph runtime, model components, and generation ecosystem on which MoEspresso is built.

License

Dual-licensed under Apache 2.0 (LICENSE-APACHE-2.0) or MIT (LICENSE-MIT), at your option.

THIRD-PARTY-NOTICES is the per-file attribution record for third-party code that ships inside this repository, with the upstream revision each derivation was taken from. It travels with the source, the wheel, and the source distribution, and a redistribution must keep all three files together.