Run Kimi-K3 (2.8T MoE) from SSD with a 25 GB RAM budget

Tiny engine, immense model. A readable, single-file C++17 inference experiment targeting consumer-memory machines. It opens Moonshot AI's official Kimi-K3 safetensors directly, keeps a bounded expert cache, and streams native MXFP4 routed experts from disk with io_uring + O_DIRECT.

The engine needs no PyTorch, CUDA, GGUF conversion, or liburing. It uses the Linux system-call ABI directly and keeps the model equations in one source file that can be read from top to bottom. The optional official-reference oracle recorder does use PyTorch on a separate, suitably large validation host.

[!IMPORTANT] This is an engineering preview, not a production inference server. Checkpoint-independent kernels and all 497,220 official tensor schemas are tested. Full 93-layer logits have not yet been compared with the official Python implementation because this development host has only one 800 GB local SSD and therefore cannot store the 1.42 TiB checkpoint. This is a storage-capacity limit, not a claim that the model needs 1.42 TiB of RAM. The only generation command is deliberately named --generate-unverified.

The consumer-memory idea

Kimi-K3 has about 2.8 trillion total parameters but activates only 16 of 896 routed experts per MoE layer. That sparsity makes a disk-backed CPU engine possible:

1. memory-map always-active attention, router, norm, embedding, and shared expert tensors;
2. keep recurrent KDA and compressed MLA state in RAM;
3. reserve a fixed number of 16.73 MiB expert-cache slots;
4. pin all selected cache hits before choosing eviction victims;
5. submit every missing expert's six tensor operands in one direct-I/O batch;
6. compute cache hits and shared experts while the batch is in flight;
7. reduce results in router order, never disk-completion order.

The bounded mutable state and minimum 16-slot expert cache are well below 25 GB. On a 25 GB machine, however, Linux must also demand-page roughly 106.5 GiB of always-active file-backed weights. It should be possible to run without loading the 1.42 TiB model into RAM, but it will be extremely storage-bound and has not yet been benchmarked on such a machine. This repository does not pretend otherwise.

You still need at least 1,638,903,341,107 free bytes for the official checkpoint plus a 5% working margin.

Build

Requirements:

• Linux x86-64;
• a C++17 compiler;
• GNU Make;
• OpenMP support;
• Python 3 only for header downloading, regression tests, and the dependency-free oracle comparator. Recording the official oracle separately requires the model's PyTorch/Transformers environment.

make
./kimik3 --self-test
./kimik3 --architecture
./kimik3 --check-runtime-state
./kimik3 --check-numa

The executable links only the ordinary C/C++ runtime, libm, and OpenMP.

Validate the official checkpoint layout without downloading 1.42 TiB

The range downloader fetches only each safetensors header plus the tokenizer:

./setup_kimik3.sh --headers /ssd/kimik3-headers

That downloads about 72 MiB of headers and then checks:

• all 96 shards;
• all 497,220 tensor records;
• every name, dtype, shape, and payload range;
• the tokenizer's pinned size and SHA-256;
• exact tokenizer and prompt vectors.

Exercise the real direct-I/O path

./kimik3 --benchmark-storage 8

The benchmark creates one temporary 17,547,264-byte expert fixture under /ssd, opens it with O_DIRECT, submits the exact six operand reads through raw io_uring, validates every byte, reports the backend actually used, and removes the fixture.

The development host reports eight io_uring batches and zero fallback batches. Throughput varies with the SSD and cache state, so the command prints its own measurement instead of baking a headline number into the project.

Use an existing checkpoint

./setup_kimik3.sh --model /ssd/Kimi-K3

Or explicitly download it after the free-space guard passes:

./setup_kimik3.sh --download /ssd/Kimi-K3

The unvalidated generation diagnostic is:

KIMIK3_EXPERT_CACHE_SLOTS=64 \
  ./kimik3 --generate-unverified \
  /ssd/Kimi-K3 /ssd/kimik3-headers/tiktoken.model \
  "Explain why sparse MoE models can stream experts from SSD." 32

Sixteen slots are the correctness minimum and consume about 268.5 MiB. Each additional aligned slot costs about 16.78 MiB.

What is implemented

• BF16 conversion and matrix-vector products.
• Native packed MXFP4 E2M1/E8M0 expert kernels.
• Exact Kimi Delta Attention recurrence and convolution state.
• Compressed latent MLA cache.
• Attention Residual block lifecycle.
• Stable LatentMoE routing, correction bias, SiTU activation, and shared experts.
• Complete stateful 93-layer single-token decoder orchestration.
• Official tiktoken BPE and complete XTML conversation rendering.
• Read-only safetensors mapping and typed tensor binding.
• Fixed-capacity deterministic LRU expert cache.
• Two-phase hit pinning and miss scheduling.
• Batched io_uring reads with O_DIRECT and exact pread fallback.
• NUMA-local or MPOL_INTERLEAVE resident tensor placement.
• Greedy, temperature, and top-p sampling.
• Versioned decoder-boundary oracle with exact route checks and numerical comparison against the pinned official implementation.

Why one C++ file?

The point is auditability. kimik3.cpp is ordered as a story:

architecture and resource accounting
→ number formats and kernels
→ KDA / MLA / Attention Residuals / MoE
→ checkpoint mapping
→ tokenizer and XTML
→ expert cache and asynchronous storage
→ 93-layer decoder
→ tests and CLI

There is no graph compiler or tensor framework between the model equations and the code executing them.

Current correctness boundary

The local tests prove the components and metadata contract. They do not prove that complete checkpoint-backed generation matches Moonshot's implementation. The oracle machinery for the remaining acceptance test is included and documented in KIMIK3_ORACLE.md. It compares:

1. embeddings;
2. pre-attention residual merge;
3. KDA or MLA output;
4. pre-MoE merge;
5. routed expert IDs and weights;
6. every layer output;
7. final logits and sampled token.

The normal generation command will stay gated until that oracle passes on a host capable of loading the complete checkpoint.

Files

• kimik3.cpp — the engine and deterministic self-tests.
• Makefile — release build and focused test targets.
• setup_kimik3.sh — build, metadata validation, and guarded download.
• fetch_kimik3_headers.py — pinned parallel range downloader.
• test_kimik3_tokenizer.py — checkpoint-independent tokenizer vectors.
• test_kimik3_xtml.py — exact multi-turn/tools/schema/image prompt vectors.
• kimik3_oracle.py — official-reference recorder and dependency-free comparator.
• KIMIK3_ORACLE.md — byte format, commands, tolerances, and acceptance plan.
• scripts/check_public_tree.py — fail-closed public-tree policy audit.

References:

• Moonshot AI Kimi-K3
• Colibri, the tiered-storage inspiration