Kernel level and memory system engineering

• AI and memory wall: How memory is the main bottleneck for LLM?
• Optimal Software Pipelining and Warp Specialization for Tensor Core GPUs by NVIDIA and Stanford University
• Collective Communication for 100k+ GPUs by Meta
• Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators by Meta
• Parallel CPU-GPU Execution for LLM Inference on Constrained GPUs
• The Landscape of GPU-Centric Communication
• Use microbenchmarking and instruction level analysis to analyze GPU architecture

Algorithms for Kernel development:

• Online normalizer calculation for softmax
• Flash Attention paper1, paper for v2, paper for v3

Training engineering

• Pre-training under infinite compute by Stanford University
• Small Batch Size Training for Language Models
• End-to-End Test-Time Training for Long Context by NVIDIA and others
• Give Me BF16 or Give Me Death by RedHat and Give Me FP32 or Give Me Death?

Others:

• LLMs don't just memorize, they build a geometric map that helps them reason by Google
• Self-Adapting Language Models by MIT

• NVIDIA Developer Blog: Deep dive into multiple AI topics.
• TensorRT LLM tech blogs: Deep dive into technical techniques/optimizations in one of the leading LLM inference library. (13 posts as of now)
• SGLang tech blog: SGLang is one of the leading LLM serving framework. Most blogs are around SGLang but is rich in technical information.
• AI System co-design at Meta

YouTube channels to follow regularly:

• vLLM office hours: Deep dive into various technical topics in vLLM
• GPU Mode: Deep dive into various LLM topics from guests from the AI community
• PyTorch channel: videos of various PyTorch events covering keynotes of technical topics like torch.compile.
• AI engineer channel

• GPU architecture
  Current SOTA AI/LLM workloads are possible only because of GPUs. Understanding GPU architecture gives you an engineering edge.

• Understanding GPU architecture with MatMul, intuition about GPUs
• GPU Shared memory banks / microbenchmarks GPU programming concepts:

• Performance

• Performance metrics by nccl tests, Profiling guide by Nsight, Understanding DL performance

• Kernel development

• Blackwell Tensor Core MMA layouts
• Programming Massively Parallel Processors series on YT
• CUDA programming model, GPU memory management: Mark Harris's GTC Talk on Coalesced Memory Access, Prefix Sum/ Scan in GPU
• GPU Triggered Networking for Intra-Kernel Communications by AMD

• ML Sys
  • Disaggregated Scheduling / Routing
  • P/D disaggregation, DistServe P/D disaggregation paper
  • Power aware dynamic reallocation for Inference by AMD
  • KVCache-centric disaggregated architecture by MooncakeAI
  • Disaggregated inference in heterogeneous environment

• OverFill: Two-Stage Models for Efficient Language Model Decoding by Cornell University

• Transformer

• CME 295: Basics of Transformer and LLM course by Stanford University
• Transformer overall: Encoder-only and Decoder-only models
• BERT (insightful): BERT as text diffusion step
• Memory requirements for LLM. There are 4 parts: activation, parameter, gradient, optimizer states. How LLM handle memory?

• Attention

• Self-attention / Multi-head attention (MHA), Multi-Query attention (MQA), Group Query Attention (GQA), MLA (used in DeepSeek)
• FlashAttention (paper1, paper for v2, paper for v3, Online softmax, Implementation by Tri Dao
• Ring Attention (links to Context Parallelism CP): Handles large sequence length, Flex Attention by PyTorch
• KV cache, FP8 KV cache, Paged Attention
• Data Parallel (DP) Attention

Core operations

• GEMM / MatMul, API of GEMM, GEMM as core of AI, W4A8 GEMM Kernel
• MoE (Mixture of experts)
• Embedding (deepdive), RoPE (paper)

• Quantization

• Neural Network Quantization guide by Qualcomm (cover PTQ, QAT)
• Quantization basics
• Different data type stimulations
• INT8 quantization using QAT, LLM quantization with PTQ, FP8 datatype, AWQ
• Per-tensor and per-block scaling
• NVFP4 training, Optimizing FP4 Mixed-Precision Inference on AMD GPUs, Recent LLM quantization progress
• Quantization on CPU (GGUF, AWQ, GPTQ), GGUF quantization method, GPTQ: Post training quantization for LLM. OBQ: Post-Training Quantization and Pruning
• Mixed precision training
• NVFP4, GEMV kernel for NVFP4
• Details on FP8 training

• Pruning and distillation

• Post-training

• Post training concepts with SFT, RLHF, RLFR
• Smol Training Playbook: The Secrets to Building World-Class LLMs

• Optimizations

• LLM Inference optimizations; optimizations v2
• 5D parallelism PP, SP, DP, TP, CP, EP, parallelism concept for LLM scaling. Parallelism in PyTorch
• Chunk prefill - SARATHI paper, dynamic and continuous batching
• KV cache offloading, KVcache early reuse
• Speculative decoding, (basic introduction), Look-ahead reasoning, Paper from Google and DeepMind
• MoE using Wide Expert Parallelism EP

• Software tools AI

• vLLM arch: architecture of the leading LLM serving engine.

Insights:

• MinMax M2 using Full Attention: why full attention is better than masked attention?

Practical:

• CUDA Compiler & PTX with example
• CUTLASS: template library to code in CUDA easily
• Matrix transpose using CUTLASS
• SGLang inference engine architecture
• FlexAttention using CuTE DSL
• MatMul using WGMMA, GEMM with pipelining in CUTLASS