Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
See the code
A hands-on tutorial on LLM inference: build a mini-sglang from scratch, then read the real SGLang source.
Start from what inference really is, build a mini-sglang by hand, then read the real SGLang source
Datawhale × RadixArk
Course status: Part 0, Part I, and Chapters 1–2 of Part II are complete in both English and Chinese. The remaining chapters of Part II and Parts III–IV are under development. You can switch to 简体中文 at any time. Want to help? See Contributing.
Large models have moved on from the training race to the inference race. Once applications took off, inference cost and latency became the real bottleneck: on the same GPU, a good inference engine can serve several times more requests. Yet there are few tutorials on inference engines. They either stay at the concept level without touching code, or drop you straight into the SGLang source with no idea where to start.
This course fills that gap. The first half starts from the core questions of inference: why KV Cache exists, how prefill differs from decode, what compute-bound and memory-bound actually mean. The middle part walks you through building a mini-sglang from scratch: forward pass, generation, KV Cache, HTTP serving, Continuous Batching, Paged KV Cache, RadixAttention, added one at a time. The last part returns to the real SGLang, explains how its cutting-edge optimizations are actually implemented, and finally shows you how to land your first solid PR in SGLang.
The project is jointly launched by Datawhale and RadixArk (the company founded by the SGLang team). SGLang is technically solid and actively iterated in the inference framework space. If you like this project, please give the official SGLang repository a ⭐.
Status legend: ✅ Done 🔄 In progress 📝 Needs polish 🚧 Planned ⏸️ Paused
Chapters in bold are written by SGLang core members.
| Chapter | Key content | Status |
|---|---|---|
| Part 0 — Before you learn | ||
| 0.1 Coding Ethics and Open-Source Spirit | Own your code, communicate like a human, profile first, the open-source spirit | ✅ |
| 0.2 Deploy Your First SGLang Server | Environment setup; run Qwen3-0.6B with SGLang on your own GPU | ✅ |
| Part I — Foundations (concepts only, no code, no GPU) | ||
| 1. Introduction to LLM | What an LLM is and how it evolved, the Transformer architecture, autoregressive generation, key concepts | ✅ |
| 2. Introduction to Inference | Training vs. inference, prefill/decode, compute-bound vs. memory-bound, the Roofline model | ✅ |
| 3. Introduction to GPU | GPU architecture basics, how LLM inference executes on a GPU, understanding inference bottlenecks from the hardware | ✅ |
| 4. KV Cache: The Core Data Structure of Inference | Deriving KV Cache from attention, cache lifecycle, quantitative memory analysis | ✅ |
| 5. Introduction to Benchmark | Core metrics such as TTFT / TPOT / ITL / Goodput, percentiles and tail latency, how to design, run and read a benchmark | ✅ |
| Part II — Build Your Own Mini SGL | ||
| 1. mini-sglang: What an Inference Engine Looks Like | Overall architecture of an inference engine, module breakdown, roadmap for this part | ✅ |
| 2. Inside SGLang: The Path of a Request | The full lifecycle of a request from arrival to response | ✅ |
| 3. Your First 200 Lines: Forward Pass and Generation | Hand-writing the forward pass and the autoregressive generation loop | 🚧 |
| 4. KV Cache: From O(n²) to O(n) | Implementing the cache and optimizing attention computation | 🚧 |
| 5. HTTP and Concurrent Requests (Concepts) · Code Walkthrough | HTTP basics and request flow; implementing isolation, streaming, and cancellation | ✅ |
| 6. Continuous Batching and the Scheduler | Continuous batching and scheduler design | 🚧 |
| 7. Paged KV Cache and Memory Management | Paged KV Cache and GPU memory management | 🚧 |
| 8. RadixAttention and Prefix Caching (Concepts) · Code Walkthrough | RadixAttention and prefix caching | ✅ |
| 9. Multi-process & Tensor Parallelism | Multi-process execution and tensor parallelism | 🚧 |
| 10. Speculative Decoding | Speculative decoding | 🚧 |
| Part III — Advanced Inference Technique (deep into the real SGLang) | ||
| 1. Attention Backends (FlashInfer / Triton / FA3 / FlashMLA) and CUDA Graph | Comparing mainstream attention backends, and CUDA Graph | 🚧 |
| 2. Quantization and Low-Precision Inference | Quantization and low-precision inference | 🚧 |
| 3. Hierarchical Caching | Hierarchical caching | 🚧 |
| 4. Scaling Out: DP Attention, EP, PP | Scaling out: DP Attention / EP / PP | 🚧 |
| 5. Prefill-Decode Disaggregation | Prefill-Decode disaggregation | 🚧 |
| Part IV — How to Make Contribution to SGLang (optional) | ||
| 1. Deploying SGLang with the Cookbook | Deploying SGLang with the Cookbook | 🚧 |
| 2. Measuring It All: Profiling and Trace Analysis | Profiling and trace analysis | 🚧 |
| 3. SGLang PR Workflow | The full SGLang PR workflow | 🚧 |
Start date: Monday, August 24, 2026. About one week per chapter; chapters in bold are written by SGLang core members.
| Part | Schedule | Status |
|---|---|---|
| Part 0 — Before you learn | Done | ✅ |
| Part I — Foundations | Writing 8.24 ~ 9.06, review 9.07 ~ 9.13 | ✅ |
| Part II — Build Your Own Mini SGL | Writing 9.14 ~ 10.04, review 10.05 ~ 10.10 | 🔄 |
| Part III — Advanced Inference Technique | 10.10 ~ 11.15 | 🚧 |
| Part IV — How to Make Contribution to SGLang | 11.16 ~ 12.6 | 🚧 |
# Clone the repository
git clone https://github.com/datawhalechina/zero-to-sglang.git
cd zero-to-sglang
# Install dependencies as each chapter requires
1️⃣ Read Part I to build an overall picture of inference (no GPU needed)
2️⃣ Follow Part II to build a mini-sglang from 0 to 1
3️⃣ Move to Part III and understand the frontier optimizations against the real SGLang source
4️⃣ Finish Part IV: profiling, trace analysis, and the full PR workflow
zero-to-sglang/
├── course-material/ # Course text
│ ├── ch/ # Chinese edition (read online at /ch/)
│ │ ├── part0/ # Part 0: Before you learn
│ │ ├── part1/ # Part I: Foundations (complete)
│ │ ├── part2/ # Part II: Build Your Own Mini SGL (in progress)
│ │ ├── part3/ # Part III: Advanced Inference Technique (planned)
│ │ ├── part4/ # Part IV: How to Make Contribution to SGLang (planned)
│ │ └── WRITING_TEMPLATE.md
│ └── eng/ # English edition (read online at /eng/, in progress)
│ ├── part0/
│ └── WRITING_TEMPLATE.md
├── community/ # Community corner: build logs, pitfalls, study notes
│ ├── ch/
│ └── eng/
├── docs/.vitepress/ # VitePress site config
├── README.md # Project overview (English, default)
├── README_zh.md # Project overview (Chinese)
├── README_en.md # Legacy English entry (links to README.md)
└── .gitignore # Git ignore rules
We are an open community and welcome contributions of every kind. Before you start, please read Part 0 to understand the coding ethics and open-source spirit we expect.
1. Improve the course text
course-material/eng/2. Share your practice
Community content goes under the matching directory in community/eng/ (English) or community/ch/ (Chinese).
How to open a PR
mainYes. Part I involves no GPU at all. Most of the code in Part II can be debugged on CPU, but full performance verification and the deeper content of Part III are best done on a GPU (cloud is fine).
No. The course starts from inference concepts. Low-level topics such as attention backends only come in Part III, and you can catch up on CUDA as needed at that point.
Join the zero-to-sglang reader community to learn together, ask questions, and share ideas: Join our Discord community
We thank every developer who has contributed to this project!
If zero-to-sglang helps your research or work, please cite it:
@misc{zero_to_sglang2026,
title = {zero-to-sglang: Building an LLM Inference Engine from Scratch},
author = {TODO},
year = {2026},
url = {https://github.com/datawhalechina/zero-to-sglang},
note = {GitHub repository}
}
If this project helps you, please give it a Star ⭐️!
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Helping more people learn to build LLM inference engines, systematically
Made with ❤️ by Datawhale & RadixArk
396 followers · starred Sep 2026
2,128 followers · starred Sep 2026
1,031 followers · starred Oct 2026
54 followers · starred Sep 2026
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
See the code
A hands-on tutorial on LLM inference: build a mini-sglang from scratch, then read the real SGLang source.
Start from what inference really is, build a mini-sglang by hand, then read the real SGLang source
Datawhale × RadixArk
Course status: Part 0, Part I, and Chapters 1–2 of Part II are complete in both English and Chinese. The remaining chapters of Part II and Parts III–IV are under development. You can switch to 简体中文 at any time. Want to help? See Contributing.
Large models have moved on from the training race to the inference race. Once applications took off, inference cost and latency became the real bottleneck: on the same GPU, a good inference engine can serve several times more requests. Yet there are few tutorials on inference engines. They either stay at the concept level without touching code, or drop you straight into the SGLang source with no idea where to start.
This course fills that gap. The first half starts from the core questions of inference: why KV Cache exists, how prefill differs from decode, what compute-bound and memory-bound actually mean. The middle part walks you through building a mini-sglang from scratch: forward pass, generation, KV Cache, HTTP serving, Continuous Batching, Paged KV Cache, RadixAttention, added one at a time. The last part returns to the real SGLang, explains how its cutting-edge optimizations are actually implemented, and finally shows you how to land your first solid PR in SGLang.
The project is jointly launched by Datawhale and RadixArk (the company founded by the SGLang team). SGLang is technically solid and actively iterated in the inference framework space. If you like this project, please give the official SGLang repository a ⭐.
Status legend: ✅ Done 🔄 In progress 📝 Needs polish 🚧 Planned ⏸️ Paused
Chapters in bold are written by SGLang core members.
| Chapter | Key content | Status |
|---|---|---|
| Part 0 — Before you learn | ||
| 0.1 Coding Ethics and Open-Source Spirit | Own your code, communicate like a human, profile first, the open-source spirit | ✅ |
| 0.2 Deploy Your First SGLang Server | Environment setup; run Qwen3-0.6B with SGLang on your own GPU | ✅ |
| Part I — Foundations (concepts only, no code, no GPU) | ||
| 1. Introduction to LLM | What an LLM is and how it evolved, the Transformer architecture, autoregressive generation, key concepts | ✅ |
| 2. Introduction to Inference | Training vs. inference, prefill/decode, compute-bound vs. memory-bound, the Roofline model | ✅ |
| 3. Introduction to GPU | GPU architecture basics, how LLM inference executes on a GPU, understanding inference bottlenecks from the hardware | ✅ |
| 4. KV Cache: The Core Data Structure of Inference | Deriving KV Cache from attention, cache lifecycle, quantitative memory analysis | ✅ |
| 5. Introduction to Benchmark | Core metrics such as TTFT / TPOT / ITL / Goodput, percentiles and tail latency, how to design, run and read a benchmark | ✅ |
| Part II — Build Your Own Mini SGL | ||
| 1. mini-sglang: What an Inference Engine Looks Like | Overall architecture of an inference engine, module breakdown, roadmap for this part | ✅ |
| 2. Inside SGLang: The Path of a Request | The full lifecycle of a request from arrival to response | ✅ |
| 3. Your First 200 Lines: Forward Pass and Generation | Hand-writing the forward pass and the autoregressive generation loop | 🚧 |
| 4. KV Cache: From O(n²) to O(n) | Implementing the cache and optimizing attention computation | 🚧 |
| 5. HTTP and Concurrent Requests (Concepts) · Code Walkthrough | HTTP basics and request flow; implementing isolation, streaming, and cancellation | ✅ |
| 6. Continuous Batching and the Scheduler | Continuous batching and scheduler design | 🚧 |
| 7. Paged KV Cache and Memory Management | Paged KV Cache and GPU memory management | 🚧 |
| 8. RadixAttention and Prefix Caching (Concepts) · Code Walkthrough | RadixAttention and prefix caching | ✅ |
| 9. Multi-process & Tensor Parallelism | Multi-process execution and tensor parallelism | 🚧 |
| 10. Speculative Decoding | Speculative decoding | 🚧 |
| Part III — Advanced Inference Technique (deep into the real SGLang) | ||
| 1. Attention Backends (FlashInfer / Triton / FA3 / FlashMLA) and CUDA Graph | Comparing mainstream attention backends, and CUDA Graph | 🚧 |
| 2. Quantization and Low-Precision Inference | Quantization and low-precision inference | 🚧 |
| 3. Hierarchical Caching | Hierarchical caching | 🚧 |
| 4. Scaling Out: DP Attention, EP, PP | Scaling out: DP Attention / EP / PP | 🚧 |
| 5. Prefill-Decode Disaggregation | Prefill-Decode disaggregation | 🚧 |
| Part IV — How to Make Contribution to SGLang (optional) | ||
| 1. Deploying SGLang with the Cookbook | Deploying SGLang with the Cookbook | 🚧 |
| 2. Measuring It All: Profiling and Trace Analysis | Profiling and trace analysis | 🚧 |
| 3. SGLang PR Workflow | The full SGLang PR workflow | 🚧 |
Start date: Monday, August 24, 2026. About one week per chapter; chapters in bold are written by SGLang core members.
| Part | Schedule | Status |
|---|---|---|
| Part 0 — Before you learn | Done | ✅ |
| Part I — Foundations | Writing 8.24 ~ 9.06, review 9.07 ~ 9.13 | ✅ |
| Part II — Build Your Own Mini SGL | Writing 9.14 ~ 10.04, review 10.05 ~ 10.10 | 🔄 |
| Part III — Advanced Inference Technique | 10.10 ~ 11.15 | 🚧 |
| Part IV — How to Make Contribution to SGLang | 11.16 ~ 12.6 | 🚧 |
# Clone the repository
git clone https://github.com/datawhalechina/zero-to-sglang.git
cd zero-to-sglang
# Install dependencies as each chapter requires
1️⃣ Read Part I to build an overall picture of inference (no GPU needed)
2️⃣ Follow Part II to build a mini-sglang from 0 to 1
3️⃣ Move to Part III and understand the frontier optimizations against the real SGLang source
4️⃣ Finish Part IV: profiling, trace analysis, and the full PR workflow
zero-to-sglang/
├── course-material/ # Course text
│ ├── ch/ # Chinese edition (read online at /ch/)
│ │ ├── part0/ # Part 0: Before you learn
│ │ ├── part1/ # Part I: Foundations (complete)
│ │ ├── part2/ # Part II: Build Your Own Mini SGL (in progress)
│ │ ├── part3/ # Part III: Advanced Inference Technique (planned)
│ │ ├── part4/ # Part IV: How to Make Contribution to SGLang (planned)
│ │ └── WRITING_TEMPLATE.md
│ └── eng/ # English edition (read online at /eng/, in progress)
│ ├── part0/
│ └── WRITING_TEMPLATE.md
├── community/ # Community corner: build logs, pitfalls, study notes
│ ├── ch/
│ └── eng/
├── docs/.vitepress/ # VitePress site config
├── README.md # Project overview (English, default)
├── README_zh.md # Project overview (Chinese)
├── README_en.md # Legacy English entry (links to README.md)
└── .gitignore # Git ignore rules
We are an open community and welcome contributions of every kind. Before you start, please read Part 0 to understand the coding ethics and open-source spirit we expect.
1. Improve the course text
course-material/eng/2. Share your practice
Community content goes under the matching directory in community/eng/ (English) or community/ch/ (Chinese).
How to open a PR
mainYes. Part I involves no GPU at all. Most of the code in Part II can be debugged on CPU, but full performance verification and the deeper content of Part III are best done on a GPU (cloud is fine).
No. The course starts from inference concepts. Low-level topics such as attention backends only come in Part III, and you can catch up on CUDA as needed at that point.
Join the zero-to-sglang reader community to learn together, ask questions, and share ideas: Join our Discord community
We thank every developer who has contributed to this project!
If zero-to-sglang helps your research or work, please cite it:
@misc{zero_to_sglang2026,
title = {zero-to-sglang: Building an LLM Inference Engine from Scratch},
author = {TODO},
year = {2026},
url = {https://github.com/datawhalechina/zero-to-sglang},
note = {GitHub repository}
}
If this project helps you, please give it a Star ⭐️!
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Helping more people learn to build LLM inference engines, systematically
Made with ❤️ by Datawhale & RadixArk
396 followers · starred Sep 2026
2,128 followers · starred Sep 2026
1,031 followers · starred Oct 2026
54 followers · starred Sep 2026