Goal: Get to a fast, working engine for a dense model as quickly as possible. Limit: A fixed candidate budget.

mtp7-fp8attn runs Qwen3.8-27B in its original NVFP4 weights on one RTX 5090, with the model's full 262,144 tokens of context. One coding agent built it from scratch in a single crit run, writing its own kernels on Core without using another inference engine as a reference. It speeds up decoding with the model's own built-in draft layer, used as released.

Measured with crit Eval at the full 262,144-token context. Decode is the geometric mean over six prompts, from a chat question to a 14K-token agent session.

• Long context: a prompt that fills all 262,144 tokens gets its first token after 69 s, then decodes at 217 tok/s. Candidate 4's 8-bit decode attention was slower in real rounds, so candidate 5 dropped it.
• Quality: perplexity stays within 0.3% of the reference, and drafting never changes the output: it is token for token what plain decoding produces.

Goal: Get to a fast, working engine for a dense model of a different architecture. Limit: A fixed candidate budget.

assistant6-fp8qk runs Gemma 4 31B-IT on one RTX 5090 with the model's full 262,144 tokens of context. Gemma mixes 50 sliding-window layers with 10 full-attention layers whose keys equal their values, and its hidden states overflow FP16, so little carries over from the Qwen engines. The NVFP4 MLPs are used as stored; the BF16 attention projections and output head are re-encoded to FP8 at load, because as stored they don't fit next to a 262K KV cache. It drafts 6 tokens ahead with Google's own assistant model, used as released.

Measured with crit Eval at the full 262,144-token context. Candidates 4 and 5 run the same code; their difference is run-to-run variation.

• Bandwidth: the verify pass reads the weights at their bandwidth floor (14.6 ms against 15.0 ms), so decode speed is set by how many drafts are accepted: 3.1 tokens per round on chat, up to 6.4 on the agent case. A prompt that fills all 262,144 tokens gets its first token after 148 s.
• Quality: decoding stays within 0.2% perplexity of the reference. The 4-bit prompt path spends most of the drift budget (+4.6% perplexity, 91% of the budget), inside it on every path.

Goal: Quickly build a fast engine for an MoE model whose weights don't fit in VRAM. Limit: A fixed candidate budget.

mtp3-expert-cache runs Qwen3.8-Flash-Next, a 102 GB mixture-of-experts checkpoint with 512 NVFP4 experts per layer, on one 32 GB RTX 5090 with the model's full 262,144 tokens of context. The weights are used as stored: attention, the shared expert and the KV cache stay on the GPU, the routed experts (about 61 GB) sit in pinned host memory, and a GPU cache of about 7,800 slots holds the hot ones. The coding agent wrote it from scratch on Core, again without another inference engine as a reference, and drafts 3 tokens ahead with the model's own MTP block, used as released.

Measured with crit Eval at the full 262,144-token context. The work took two crit runs back to back, from 14:37 to 04:34. The first run's request asked for 1 GiB of free VRAM instead of the machine's 0.3 GiB, so none of its candidates qualified, and the second continued from its candidate 2. Candidate 5 was selected first, but an audit found it reused prefetched experts before their scales were initialized, so generation after prompts of 512 or more tokens was wrong; Eval's prompts and scoring paths never decoded after such a prompt. Candidate 6 fixes it and replaces candidate 5 in crit-hub.

• Bottleneck: decode is bound by expert misses over the host link (about 27 GB/s): each round reads 88 to 280 missed experts. A prompt that fills all 262,144 tokens gets its first token after about 100 s, then decodes at 63 tok/s.
• Quality: perplexity stays within 0.03% of the reference. Drafted output equals plain decoding on 3 of 6 cases; the cause is not yet established, and this run reported exactness rather than requiring it.

Goal: Build a fast engine for a 120B hybrid Mamba and mixture-of-experts model whose weights don't fit in VRAM. Limit: A fixed candidate budget.

mtp3-cpu-experts runs Nemotron 3.5 Super VL 120B-A12B (text path) on one RTX 5090 with 262,144 tokens of context. Its layers are 40 Mamba-2 mixers, 40 latent mixture-of-experts layers with 512 NVFP4 experts each (22 per token) and 8 attention layers. The checkpoint is 75 GB, 63 GB of it routed experts, used as stored: they sit in pinned host memory behind a GPU cache of about 7,150 slots, and 6 CPU cores compute 60% of each pass's cache misses with the GPU's exact operation order, so the output does not depend on where a miss runs. It drafts 3 tokens ahead with the model's own MTP block, used as released.

Measured with crit Eval. Candidates 1 and 2 were built for a 1,048,576-token context; the work then moved to a second crit run at 262,144 tokens, which re-measured candidate 3 and built candidate 4. Candidates 3 and 4 differ only in the prompt path, so their decode difference is run-to-run variation. The development time leaves out about 5h 25m with no work in progress: an idle stretch in the night, and a PC restart (an engine and the FP32 reference together had exhausted host memory) followed by a pause until the work was resumed.

• Bottleneck: decode is bound by expert misses, over the host link and on the CPU, so other heavy programs on the PC slow it. Sampled requests decode one token per pass, about 22 tok/s.
• Quality: perplexity stays within 0.7% of the reference on every path. Candidate 1 lost track of content beyond 64K tokens because transformers clamps Mamba's time step; the engine and Eval's reference now follow the model's config, and retrieval passes up to 262,144 tokens.

Core provides what every engine needs: request scheduling, memory tracking, output delivery and safe cleanup. Your engine owns the model, tokenizer and layouts, and calls Core's libraries for sampling, state cache and model files. The native runtime supports generation and scoring, with optional CUDA and concurrent requests. Try the Core + Inference walkthrough.

Inference holds techniques with explanations, implementation code and worked examples. Start from a bottleneck:

• Reuse prompt work. Prefix checkpoints resume from stored state when a prompt shares an earlier prefix.
• Keep useful weights close. Expert residency retains reusable expert weights within a memory budget.
• Generate several tokens per round. Speculative decoding proposes a continuation, verifies it and keeps the accepted tokens.
• Reuse GPU execution. Core's optional CUDA graph harness captures repeated work for replay to cut launch overhead.

Each technique says when it helps, what is tested and what your engine supplies.

Kernels helps you find and assess existing kernels for your data format and hardware, including the Hugging Face Kernel Hub. It checks declared compatibility, shows mismatches and gaps, and retains chosen artifacts with exact versions and license notices. Your engine then measures whether the kernel helps.

Models produces transformed weights when a run allows them, starting with post-training quantization to FP8 or NVFP4 (round-to-nearest or GPTQ). It exports a checkpoint engines can read, runs it in your engine through Eval against the original model, and records the recipe, calibration data, seed, file hashes and cost.

Eval gives engines a common set of tests: fixed prompts, output lengths and a shared quality reference per model.

• Speed on chat, code, math, tool calling, long-document summarization and a long agent session, with decode, prefill and time-to-first-token scores against their physical ceilings.
• Quality: the engine's predictions against the reference on fixed conversations, across its execution paths.
• Speculative checks: drafting against plain greedy decoding, exact when the run requires it.
• Repeatability: one pinned decode text per model, and paired A/B runs for small changes.
• Profiling: kernels ranked per phase by GPU time and launch width.

Results keep samples, settings and skipped cases visible.

crit-hub collects crit-based inference engines in one repository to make discovery super easy. Each engine ships runnable source, the exact checkpoint it was built on, what was measured, its known limits and a link to the repository of whoever contributed it.

crit serve loads a sealed candidate, keeps it resident and answers OpenAI-compatible chat requests, including streaming, stop strings and structured output. Engines implement the adapter contract.