guqiong96/Lvllmds4-x

CPU-GPU hybrid inference for DeepSeek-V4 on NVIDIA SM80+ (A100/RTX 4090 etc.), forked from yhfgyyf/vllm-deepseek-v4-sm89.

stars

14,339

commits

Python

primary language

Sep 11, 2026

updated

space.bilibili.com/625367168 Browse cluster: LLM Serving and Inference Optimization →

README

Lvllmds4-x

A fork of yhfgyyf/vllm-deepseek-v4-sm89 with CPU-GPU hybrid inference support for DeepSeek-V4 on NVIDIA SM80+ (A100, RTX 4090, etc.).

Origin

This project integrates the CPU-GPU hybrid inference engine lk_moe into a specialized vLLM fork:

Base vLLM Fork: Forked from yhfgyyf/vllm-deepseek-v4-sm89, which provides the compatibility modifications needed for DeepSeek-V4 on SM89 architecture, extended here to support SM80+ GPUs.
Hybrid Inference: The lk_moe engine enables MOE layers to leverage both GPU VRAM and CPU system memory for collaborative computation, with NUMA-aware scheduling and expert weight management.

This version is purpose-built to run DeepSeek V4 on NVIDIA GPUs with SM80+ compute capability (A100, RTX 4090, A6000, etc.).

Relationship with LvLLM

Lvllmds4-x is part of the LvLLM ecosystem—a family of parallel projects that integrate the lk_moe hybrid inference engine into different vLLM branches for different model/hardware targets:

Project	Upstream vLLM Branch	Target
LvLLM	Latest vLLM mainline	General MoE models (Qwen3, GLM, MiniMax, Kimi, etc.)
Lvllmds4	`jasl/vllm` (`codex/ds4-sm120-min-enable`)	DeepSeek-V4 (SM120+)
Lvllmds4-x (this repo)	`yhfgyyf/vllm-deepseek-v4-sm89`	DeepSeek-V4 (SM80+)

Similarly, Lsglang integrates lk_moe into sglang for the same hybrid inference capabilities across frameworks.

Performance Benchmark

Open GPU Prefill, max_num_batched_tokens=8192 (Row 1), max_num_batched_tokens=32768 (Row 2)

Model	Version	CPU	Memory	GPU	Prefill	Decode	Speculative Decoding
deepseek-ai/DeepSeek-V4-Flash-0731	Lvllmds4-x-v2.3.9	EPYC 7642 *2	16 channels ddr4 3200	3090 * 2	1060 t/s [input 32768]	26 t/s [input 32768]	35~47 t/s
deepseek-ai/DeepSeek-V4-Flash-0731	Lvllmds4-v2.3.9	EPYC 9684x *2	24 channels ddr5 4800	pro 6000 * 1	3100 t/s [input 131072]	75 t/s [input 131072]	100~115 t/s

Usage Guide

Pre-built releases and detailed installation/usage instructions for DeepSeek-V4 on SM80+ are available on the Releases page.

Contributors

(top 30 of 459)

896 commits

799 commits

537 commits

501 commits

Similar repos

guqiong96/Lvllmds4

A fork of jasl/vllm (codex/ds4-sm120-min-enable) with CPU-GPU hybrid inference support for…

shared contributors

yhfgyyf/vllm-deepseek-v4-sm89

Run DeepSeek-V4-Flash / Vision-Exp and GLM-5.3-Flash on SM89 (Ada / RTX 4090) and SM120 (RTX PRO…

151

shared contributors

1CatAI/1Cat-vLLM

V100 / SM70-focused vLLM engineering fork for modern LLM inference.

974

shared contributors

wtdcode/vllm-backport

Dogfooding vLLM backport for older GPUs.

190

shared contributors

vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

91,518

shared contributors

guqiong96/Lsglang

Lsglang is a special extension of sglang that fully utilizes CPU and GPU computing resources with…

124

shared contributors

charlie12345/vLLM_for_AMD

vLLM optimized for AMD

shared contributors

tenstorrent/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

shared contributors

Languages

Python

84.5%

Cuda

5.3%

Rust

5.1%

C++

3.6%

guqiong96/Lvllmds4-x

CPU-GPU hybrid inference for DeepSeek-V4 on NVIDIA SM80+ (A100/RTX 4090 etc.), forked from yhfgyyf/vllm-deepseek-v4-sm89.

stars

14,339

commits

Python

primary language

Sep 11, 2026

updated

space.bilibili.com/625367168 Browse cluster: LLM Serving and Inference Optimization →

README

Lvllmds4-x

A fork of yhfgyyf/vllm-deepseek-v4-sm89 with CPU-GPU hybrid inference support for DeepSeek-V4 on NVIDIA SM80+ (A100, RTX 4090, etc.).

Origin

This project integrates the CPU-GPU hybrid inference engine lk_moe into a specialized vLLM fork:

Base vLLM Fork: Forked from yhfgyyf/vllm-deepseek-v4-sm89, which provides the compatibility modifications needed for DeepSeek-V4 on SM89 architecture, extended here to support SM80+ GPUs.
Hybrid Inference: The lk_moe engine enables MOE layers to leverage both GPU VRAM and CPU system memory for collaborative computation, with NUMA-aware scheduling and expert weight management.

This version is purpose-built to run DeepSeek V4 on NVIDIA GPUs with SM80+ compute capability (A100, RTX 4090, A6000, etc.).

Relationship with LvLLM

Lvllmds4-x is part of the LvLLM ecosystem—a family of parallel projects that integrate the lk_moe hybrid inference engine into different vLLM branches for different model/hardware targets:

Project	Upstream vLLM Branch	Target
LvLLM	Latest vLLM mainline	General MoE models (Qwen3, GLM, MiniMax, Kimi, etc.)
Lvllmds4	`jasl/vllm` (`codex/ds4-sm120-min-enable`)	DeepSeek-V4 (SM120+)
Lvllmds4-x (this repo)	`yhfgyyf/vllm-deepseek-v4-sm89`	DeepSeek-V4 (SM80+)

Similarly, Lsglang integrates lk_moe into sglang for the same hybrid inference capabilities across frameworks.

Performance Benchmark

Open GPU Prefill, max_num_batched_tokens=8192 (Row 1), max_num_batched_tokens=32768 (Row 2)

Model	Version	CPU	Memory	GPU	Prefill	Decode	Speculative Decoding
deepseek-ai/DeepSeek-V4-Flash-0731	Lvllmds4-x-v2.3.9	EPYC 7642 *2	16 channels ddr4 3200	3090 * 2	1060 t/s [input 32768]	26 t/s [input 32768]	35~47 t/s
deepseek-ai/DeepSeek-V4-Flash-0731	Lvllmds4-v2.3.9	EPYC 9684x *2	24 channels ddr5 4800	pro 6000 * 1	3100 t/s [input 131072]	75 t/s [input 131072]	100~115 t/s