Lsglang — lk_moe Hybrid Inference for sglang [中文]

Lsglang is a special extension of sglang that adds CPU-GPU hybrid (MOE) inference on top of the latest sglang release version, fully compatible with stock sglang.

The actual hybrid inference engine is lk_moe, sglang/vllm only provide the "GPU path", lk_moe provides the "hybrid path". Lsglang is the concrete integration case of lk_moe into sglang.

Release policy: Lsglang version updates are released in sync with sglang releases — on top of a fresh sglang tag we keep the code "as-is + lk_moe". We do not pile on extra features; unless a necessary bug-fix patch is required, the diff against upstream stays minimal (just the lk_moe layer).

---

一、Why lk_moe?

lk_moe lets the MOE model footprint span VRAM + system memory, and schedules expert computation across CPU + GPU with NUMA awareness:

• VRAM + Memory load balancing: total footprint = VRAM + memory, so a model can be "1+1=2" and reach 100% VRAM utilization.
• CPU-GPU hybrid decode / prefill + GPU prefill: three computing modes, with GPU prefill running in parallel with hybrid decoding for near-100% GPU utilization.
• NUMA thread optimization: cross-node communication as low as 3%, L3 cache hit rate over 50%.

Note 1: x86 CPUs with AVX2+ instruction sets and Nvidia GPUs with sm80+ architectures.

---

二、How to integrate lk_moe

lk_moe is a pip-installable package (pip install lk_moe). It exposes a small set of C++ kernel classes (MOE_WNA16, MOE_FP8, MOE_MXFP4, LKEmbedding, ...) driven by a MOEConfigV2 config. The engine handles expert weight placement (VRAM / pinned NUMA host memory), NUMA-aware scheduling, and quantized kernel execution internally.

The integration work in sglang/vllm is therefore only about routing each MOE layer to lk_moe (which layers stay on GPU, which go hybrid, which quant kernel to use) and keeping the feature optional so the branch stays 100% compatible with stock behavior when disabled.

Core integration principle

Every MOE layer can be one of three roles. The role is decided by a few env vars, and the rest of the engine is unchanged.

Minimal integration checklist

1. Add the dependency — lk_moe in python/pyproject.toml (for sglang) / requirements (for vllm).
2. Add a feature gate — is_lk_moe_feature_enabled() (reads LVLLM_MOE_NUMA_ENABLED) so all hybrid behavior is off by default and the branch behaves exactly like stock sglang/vllm.
3. Wire the MOE layer — in the fused-MoE layer, resolve each layer's role, build a lk_moe.MOEConfigV2, instantiate the quant-appropriate MOE_* class, and call it in forward.
4. Register per-quantization kernels — each quant method exposes its own LK MoE kernel class.
5. Handle weight loading / placement — keep CPU-resident weights off the GPU device.
6. (Optional) extras — CPU-resident embedding (LKEmbedding) and NUMA thread binding.

Case study — Lsglang (sglang) file-by-file

The whole lk_moe integration is captured as a single portable patch at patches/01_lk_moe__v0.5.19.patch — the full diff between upstream v0.5.19 and the merge commit 45101ca52 "Merge v0.5.19 into lk_moe branch". Apply it to a clean v0.5.19 checkout with git apply patches/01_lk_moe__v0.5.19.patch.

LvLLM (vllm)

The same method is applied to vLLM in the Lvllm repository (vllm model_executor/layers/fused_moe, quantization, model_loader), plus dedicated DeepSeek-V4 branches: Lvllmds4 (SM120+) and Lvllmds4-x (SM80+).

---

三、Example — Lsglang (with benchmarks)

Performance benchmark

Open GPU Prefill, max_num_batched_tokens=8192 (row 1) / 32768 (row 2):

Version history

2026-09-07: Lsglang-v1.5.0 - sglang v0.5.19 + lk_moe v2.4.2 + DeepSeek V4 SM80+ support
2026-07-08: Lsglang-v1.4.1 - add ModelOpt W4A16 NVFP4 quantization types, e.g. nvidia/GLM-5.2-NVFP4
2026-07-05: Lsglang-v1.4.0 - GPU prefill speed, CPU AVX512 opt, removed LVLLM_GPU_RESIDENT_MOE_EXPERTS, sglang v0.5.14
2026-06-05: Lsglang-v1.3.0 - upgraded lk_moe, supports nvfp4/mxfp4, added LVLLM_GPU_RESIDENT_MOE_EXPERTS
2026-04-06: Lsglang-v1.2.0 - LK_POWER_SAVING=1, FP8+BF16+AWQ4bit mixed MOE layer inference
2026-04-03: Lsglang-v1.1.4 - local sgl-kernel compilation to fix known issues
2026-03-11: Lsglang-v1.1.3 - FP8/AWQ4bit no extra memory with GPU prefill
2026-03-05: Lsglang-v1.1.0 - GPU prefill support
2026-02-25: Lsglang-v1.0.6 - bug fixes, new models
2026-02-10: Lsglang-v1.0.0 - ported from LvLLM; verified BF16/F16, FP8, AWQ 4bit

Supported models & quant formats

Most original MOE models verified on Lsglang (Qwen3/GLM/MiniMax series etc.): gemma-4-26B-A4B-it, NVIDIA-Nemotron-3-Super-120B-A12B-BF16, Qwen3.6/3.5-35B-A3B, Qwen3.5-122B-A10B, Qwen3.5-397B-A17B, Qwen3-Coder-Next / 30B-A3B, Qwen3-VL-30B, MiniMax-M2.7/2.5/2.1, GLM-5.2-NVFP4, GLM-5.1/5.0-FP8, GLM-4.7(-Flash)/4.6V, Kimi k2.6/k2.5, deepseek-ai/DeepSeek-V4-Flash-0731 [sm80+].

Quantization formats supported at runtime: bfloat16 / float16, fp8, nvfp4, mxfp4, awq 4bit symmetric (w4a16). AWQ models: https://hf-mirror.com/cyankiwi

Quick start (DeepSeek V4 Flash [RTX 3090 *2 OR 5060Ti *2])

LVLLM_MOE_NUMA_ENABLED=1 \
LK_THREAD_BINDING=CPU_CORE \
LK_THREADS=44 \
OMP_NUM_THREADS=44 \
LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=2048 \
LVLLM_GPU_PREFETCH_WINDOW=1 \
LVLLM_GPU_RESIDENT_MOE_LAYERS=0-1,33-34 \
LVLLM_ENABLE_NUMA_INTERLEAVE=1 \
LVLLM_ENABLE_MOE_LAYERWISE_LOAD=1 \
python -m sglang.launch_server \
    --model /home/guqiong/Downloads/DeepSeek-V4-Flash-0731 \
    --served-model-name DeepSeek-V4-Flash \
    --host 0.0.0.0 --port 8070 \
    --trust-remote-code \
    --tensor-parallel-size 2 \
    --max-running-requests 2 \
    --chunked-prefill-size 32000 \
    --max-total-tokens 66000 \
    --mem-fraction-static 0.90 \
    --disable-shared-experts-fusion

Configuration parameters

Installation

conda create -n Lsglang python==3.12.11 && conda activate Lsglang
conda install -c conda-forge libstdcxx-ng
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
sudo apt-get install libnuma-dev      # Ubuntu  /  sudo dnf install numactl-devel  # Rocky

pip install lsglang                   # or build from source below

From source:

git clone https://github.com/guqiong96/Lsglang.git
cd Lsglang
pip install -U setuptools wheel scikit-build-core cmake
pip install torchaudio triton torchvision torch==2.13.0
pip install grpcio-tools wheel-stub
MAX_JOBS=32 NVCC_THREADS=1 CMAKE_BUILD_TYPE=Release \
CMAKE_ARGS="-DCMAKE_BUILD_TYPE=Release" \
pip install -e "python" --no-build-isolation -vvv

(MAX_JOBS=32 NVCC_THREADS=1: reduce compile memory; CMAKE_BUILD_TYPE=Release: perf option.)

Release / packaging example

The Lsglang release workflow is a plain editable-install + wheel build + upload:

# clean any previous build artifacts
rm -rf python/build dist

# arch list covering the supported GPUs (Ampere sm75/sm80/sm86/sm89,
# Hopper sm90, Blackwell sm100/sm120)
export TORCH_CUDA_ARCH_LIST="7.5 8.0 8.6 8.9 9.0 10.0 12.0"

# editable install to verify, then build the wheel
CMAKE_BUILD_TYPE=Release CMAKE_ARGS="-DCMAKE_BUILD_TYPE=Release" \
  pip install -e "python" --no-build-isolation -vvv
CMAKE_BUILD_TYPE=Release CMAKE_ARGS="-DCMAKE_BUILD_TYPE=Release" \
  pip wheel ./python --no-build-isolation -v --wheel-dir=dist

# upload to PyPI
python -m twine upload dist/lsglang*-any*.whl --verbose

Optimization

• MoE resident in VRAM: LVLLM_GPU_RESIDENT_MOE_LAYERS=0-5 (format 0,1,8-9; some models start at non-zero layer, e.g. Step-3.5-Flash at layer 3).
• Enable GPU prefill: LVLLM_GPU_PREFETCH_WINDOW=1, LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=4096, --chunked-prefill-size 32000.
• Disable GPU prefill: LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=0, --chunked-prefill-size 4096.
• Thread binding: LK_THREAD_BINDING=CPU_CORE (best), NUMA_NODE (fixes extreme issues on virtualization / multi-instance).
• BIOS NUMA: AMD EPYC NPS4 / Intel XEON SNC4; use 2,4,8 nodes (multiple of GPU count is best), up to 32.
• Thread count: HT on → physical cores ÷ GPUs; HT off → (physical cores-2) ÷ GPUs.
• VRAM: --chunked-prefill-size drives max-batch VRAM usage.
• CPU power saving: LK_POWER_SAVING=1.

---