~1300 line, minimal, extensible LLM inference engine built from scratch.
Python
491
12 commits
updated Oct 11, 2026
SimpleLLM is an inference engine for OpenAI's gpt-oss-20b and gpt-oss-120b on a single NVIDIA H100. It is 1,351 lines of Python: llm.py (649 lines) is the engine, model/ (405) is the model and tokenizer, and kernels.py (297) holds every GPU kernel, written in Triton.
It loads the official MXFP4 checkpoints without re-quantizing them, and its GPQA-Diamond and AIME 2025 scores match OpenAI's published numbers.
Compared with vLLM 0.31.0 tuned for each workload, SimpleLLM decodes 11-15% faster at batch 1 and batch 64 on both models, and 6-16% faster at 131k context. vLLM is faster at time to first token on long prompts, at serving 120b when all requests arrive at once, and at 1,000 concurrent requests; see What's Next. Method and every run: docs/performance.md.
Requirements: one NVIDIA H100 (80 GB), CUDA 12.8+, Python 3.12. The only GPU dependencies are PyTorch, Triton and the prebuilt flash-attn wheel (used for prefill attention).
./setup.sh ./gpt-oss-120b # Python 3.12 venv, torch 2.9.1 (cu128), flash-attn 2.8.3, downloads gpt-oss-120b
source ./.venv/bin/activate
hf download openai/gpt-oss-20b --local-dir ./gpt-oss-20b --exclude "original/*" --exclude "metal/*" # optional: the 20b model
from llm import LLM
engine = LLM("./gpt-oss-20b") # or "./gpt-oss-120b"
outputs = engine.generate(["What is the meaning of life?"], max_tokens=100).result()
print(outputs[0].text)
engine.stop()
The full API (constructor options, chat, sampling settings, output fields) is in docs/api.md; more examples are in cookbook/.
Each change below was measured end to end when it landed. Details, and the ideas that did not work: docs/optimizations.md.
| change | gain |
|---|---|
| MXFP4 MoE on tensor cores at any batch size (weights reordered once at load time) | MoE 1.33x at batch 1, 1.10x at batch 64 |
| one decode-attention kernel for sinks, sliding window, GQA and the KV append (replaces four ops) | +33% at batch 1, +36% at batch 64 |
| greedy argmax and next-step inputs computed inside the CUDA graph | +20% at batch 64 |
| RoPE as one fused kernel at every batch size | +8% at batch 64 |
| routing writes its tile table per expert instead of searching a grid | 24.8 → 3.1 µs per call, +9.6% at 120b batch 64 |
| fixed decode tile shapes instead of per-process autotuning | +3.4% at 120b batch 64, and no run-to-run variance |
| CUDA graphs for any set of active slots, all captured at startup | serving 1.56x |
| paged KV cache for full-attention layers | 120b serves 64 requests instead of 19, serving 1.47x |
| ring buffer for sliding-window layers | KV per 131k sequence halved, 2x the sequences at 131k |
| MoE token tile grows with tokens per expert during prefill | 4k-token prefill MoE 236 → 151 ms, 120b serving +12.6% |
ignore_eos (every sequence produces exactly the requested tokens), warmup, CUDA graphs, each engine in its own process.<|return|> instead of the second <|end|> (cuts off some replies with two reasoning segments)OpenAI/gpt-oss-120b tensor parallelism on 8x H100sApache 2.0. The MXFP4 weight layout follows the Hopper path of OpenAI's open-source MoE kernels (MIT); see the comment in kernels.py.
3,502 followers · starred Jan 2026
4,233 followers · starred Jan 2026
7,683 followers · starred Jan 2026
61 followers · starred Jan 2026
~1300 line, minimal, extensible LLM inference engine built from scratch.
Python
491
12 commits
updated Oct 11, 2026
SimpleLLM is an inference engine for OpenAI's gpt-oss-20b and gpt-oss-120b on a single NVIDIA H100. It is 1,351 lines of Python: llm.py (649 lines) is the engine, model/ (405) is the model and tokenizer, and kernels.py (297) holds every GPU kernel, written in Triton.
It loads the official MXFP4 checkpoints without re-quantizing them, and its GPQA-Diamond and AIME 2025 scores match OpenAI's published numbers.
Compared with vLLM 0.31.0 tuned for each workload, SimpleLLM decodes 11-15% faster at batch 1 and batch 64 on both models, and 6-16% faster at 131k context. vLLM is faster at time to first token on long prompts, at serving 120b when all requests arrive at once, and at 1,000 concurrent requests; see What's Next. Method and every run: docs/performance.md.
Requirements: one NVIDIA H100 (80 GB), CUDA 12.8+, Python 3.12. The only GPU dependencies are PyTorch, Triton and the prebuilt flash-attn wheel (used for prefill attention).
./setup.sh ./gpt-oss-120b # Python 3.12 venv, torch 2.9.1 (cu128), flash-attn 2.8.3, downloads gpt-oss-120b
source ./.venv/bin/activate
hf download openai/gpt-oss-20b --local-dir ./gpt-oss-20b --exclude "original/*" --exclude "metal/*" # optional: the 20b model
from llm import LLM
engine = LLM("./gpt-oss-20b") # or "./gpt-oss-120b"
outputs = engine.generate(["What is the meaning of life?"], max_tokens=100).result()
print(outputs[0].text)
engine.stop()
The full API (constructor options, chat, sampling settings, output fields) is in docs/api.md; more examples are in cookbook/.
Each change below was measured end to end when it landed. Details, and the ideas that did not work: docs/optimizations.md.
| change | gain |
|---|---|
| MXFP4 MoE on tensor cores at any batch size (weights reordered once at load time) | MoE 1.33x at batch 1, 1.10x at batch 64 |
| one decode-attention kernel for sinks, sliding window, GQA and the KV append (replaces four ops) | +33% at batch 1, +36% at batch 64 |
| greedy argmax and next-step inputs computed inside the CUDA graph | +20% at batch 64 |
| RoPE as one fused kernel at every batch size | +8% at batch 64 |
| routing writes its tile table per expert instead of searching a grid | 24.8 → 3.1 µs per call, +9.6% at 120b batch 64 |
| fixed decode tile shapes instead of per-process autotuning | +3.4% at 120b batch 64, and no run-to-run variance |
| CUDA graphs for any set of active slots, all captured at startup | serving 1.56x |
| paged KV cache for full-attention layers | 120b serves 64 requests instead of 19, serving 1.47x |
| ring buffer for sliding-window layers | KV per 131k sequence halved, 2x the sequences at 131k |
| MoE token tile grows with tokens per expert during prefill | 4k-token prefill MoE 236 → 151 ms, 120b serving +12.6% |
ignore_eos (every sequence produces exactly the requested tokens), warmup, CUDA graphs, each engine in its own process.<|return|> instead of the second <|end|> (cuts off some replies with two reasoning segments)OpenAI/gpt-oss-120b tensor parallelism on 8x H100sApache 2.0. The MXFP4 weight layout follows the Hopper path of OpenAI's open-source MoE kernels (MIT); see the comment in kernels.py.
3,502 followers · starred Jan 2026
4,233 followers · starred Jan 2026
7,683 followers · starred Jan 2026
61 followers · starred Jan 2026