Easy, fast, and cheap omni-modality model serving for everyone

| Documentation | DeepWiki | User Forum | Developer Slack | WeChat | Paper | Slides |

---

Latest News 🔥

• [2026/07] We released 0.24.0 - aligned with the vLLM 0.24 release line, expanding production-ready coverage across TTS, speech, diffusion, image/video generation, and robot-policy serving, with major Omni stage runtime refactoring, diffusion request-level batching, async output materialization, quantization/cache/memory improvements, and broad CUDA/ROCm/XPU/NPU support.
• [2026/06] We released 0.22.0 - an omnimodal world-model release aligned with vLLM 0.22, featuring Nvidia Cosmos3/DreamZero world model support, expanded quantization coverage across Blackwell/NPU/XPU, TTS production improvements, new models including MiniCPM-o 4.5, MOSS-TTS, and Lance, plus RL integration with VeRL-Omni.
• [2026/05] We released 0.20.0 - refreshes the serving/runtime stack for large-scale omni workloads, and improves diffusion model performance, quantization, and hardware readiness across CUDA, ROCm, MUSA, NPU, and XPU backends.
• [2026/03] We released 0.18.0 - strengthens the core runtime through a large entrypoint refactor and scheduler/runtime cleanups, expands unified quantization and diffusion execution, broadens multimodal model coverage, and improves production readiness across audio, omni, image, video, RL, and multi-platform deployments.
• [2026/03] Check out our first public project deepdive at the vLLM Hong Kong Meetup!
• [2026/03] vllm-omni-skills is a community-driven collection of AI assistant skills that help developers work with vLLM-Omni more effectively. These skills can be used with popular agentic AI coding assistants like Cursor IDE, Claude, Codex, and more.
• [2026/02] We released 0.16.0 - A major alignment + capability release that rebases onto upstream vLLM v0.16.0 and significantly expands performance, distributed execution, and production readiness across Qwen3-Omni / Qwen3-TTS, Bagel, MiMo-Audio, GLM-Image and the Diffusion (DiT) image/video stack—while also improving platform coverage (CUDA / ROCm / NPU / XPU), CI quality, and documentation.
• [2026/02] We released 0.14.0 - This is the first stable release of vLLM-Omni that expands Omni’s diffusion / image-video generation and audio / TTS stack, improves distributed execution and memory efficiency, and broadens platform/backend coverage (GPU/ROCm/NPU/XPU). It also brings meaningful upgrades to serving APIs, profiling & benchmarking, and overall stability. Please check our latest paper for architecture design and performance results.
• [2025/11] vLLM community officially released vllm-project/vllm-omni in order to support omni-modality models serving.

---

About

vLLM was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:

• Omni-modality: Text, image, audio, video, and action data processing
• Non-autoregressive Architectures: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
• Heterogeneous outputs: from traditional text generation to multimodal and action outputs

vLLM-Omni is fast with:

• State-of-the-art AR support by leveraging efficient KV cache management from vLLM
• Pipelined stage execution overlapping for high throughput performance
• Fully disaggregation based on OmniConnector and dynamic resource allocation across stages

vLLM-Omni is flexible and easy to use with:

• Heterogeneous pipeline abstraction to manage complex model workflows
• Seamless integration with popular Hugging Face models
• Tensor, pipeline, data and expert parallelism support for distributed inference
• Streaming outputs
• OpenAI-compatible API server

vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:

• Omni-modality models (e.g. Qwen3-Omni, Cosmos3, HunyuanImage, BAGEL)
• TTS models (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3)
• Diffusion models — image, video, and audio generation (e.g. Qwen-Image, Wan2.2, FLUX)
• Robot-policy and action models (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy)

MiniCPM-o 4.5 Ascend NPU optimization snapshot

This fork contains an experimental MiniCPM-o 4.5 optimization snapshot for Ascend NPU. The public snapshot is anchored at commit a3d7ea489d4a5cc2d682456fa746443536b1ec67, whose source tree is identical to 4b29737eaf2c62c521450eab6dd1f4fe8b440d60. It is a research and competition artifact rather than an official upstream vLLM-Omni release.

What is included

• The complete vLLM-Omni source snapshot used by the experiment.
• A maintainer-trained, one-step CFG-free Flow student checkpoint at vllm_omni/model_executor/models/minicpmo_4_5/flow_students/cfgfree_t1/flow.pt. The maintainer trained this checkpoint and intentionally includes it in the public release through Git LFS.
• The checkpoint manifest and training report next to the checkpoint.

After cloning, install Git LFS and materialize the checkpoint:

git lfs install
git lfs pull
sha256sum vllm_omni/model_executor/models/minicpmo_4_5/flow_students/cfgfree_t1/flow.pt

The expected checkpoint is 623,362,588 bytes with SHA-256:

e0300c8aca57d129b849dca95f7d81dd2a9f6c1e2cf5e7d3623539349a5e162b

What is not included

The following items are required for the recorded benchmark but are not redistributed by this repository:

• The MiniCPM-o-4_5 base model and its tokenizer/configuration files.
• The Seed-TTS evaluation dataset.
• vLLM, vLLM-Ascend, the Ascend driver/toolkit, and a working NPU runtime.

Obtain the base model and dataset from their official distributions and comply with their respective licenses and access terms. Set local model and dataset paths in your launch and benchmark commands; the /workspace/... paths used in the original run were machine-local and are not portable.

Environment used for the recorded result

The mean_audio_rtf = 0.07949533736173581 result was recorded on the following reference environment:

These versions describe the validated machine; they are not a claim that other version combinations are unsupported. The original run emitted a vLLM / vLLM-Omni development-version alignment warning, so exact-commit reproduction is recommended before attempting upgrades.

Fast-path settings used for RTF 0.079

The 0.079 result is not the default launch result. The service used these additional environment variables:

export TASK_QUEUE_ENABLE=1
export OMP_NUM_THREADS=1
export VLLM_MINICPMO_STAGE1_CODEC_TOP_K=1
export VLLM_MINICPMO_STAGE1_GREEDY_DIRECT_LOGITS=1
export VLLM_MINICPMO_STAGE1_C1_GRAPH_SAMPLER=1
export VLLM_MINICPMO_STAGE1_C1_GRAPH_SAMPLER_LOOP=1
export VLLM_MINICPMO_STAGE1_C1_GRAPH_SAMPLER_BORROW_OUTPUT=1
export VLLM_MINICPMO_STAGE1_RUNNER_LOCAL_K=12

The benchmark used Seed-TTS Chinese, 32 prompts, concurrency 1, two warm-up requests, file-based reference audio, and SEED_TTS_SIM_EVAL=1. The recorded audio metrics were:

With the same source tree but without the additional Stage 1 fast-path variables, the corresponding quick run measured mean audio RTF 0.08555489066708713. Results can vary with NPU model, driver/toolkit versions, model assets, dataset sampling, system load, and deployment configuration.

Deployment configuration caveat

The recorded 0.079 run used deploy/minicpmo_4_5.yaml from vLLM-Omni commit a964efc55b6c36ed6a9214a8cf4bb131f368183d, with SHA-256:

b3c35aad87ddeba64781b3833b64c4a7132eaf88fc29002bb018e6b84fdb76ed

That file is not byte-identical to the default deploy YAML at a3d7ea489. Verify the deployment file hash when comparing results. A branch checkout alone provides the complete optimized source, but exact benchmark reproduction additionally requires the external model, dataset, matching runtime stack, the tested deployment configuration, and the fast-path settings above.

Getting Started

Visit our documentation to learn more.

• Installation
• Quickstart
• List of Supported Models
• Deployment Recipes for vLLM-Omni model serving

Contributing

We welcome and value any contributions and collaborations. Please check out Contributing to vLLM-Omni for how to get involved.

Citation

If you use vLLM-Omni for your research, please cite our paper:

@article{yin2026vllmomni,
  title={vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models},
  author={Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu},
  journal={arXiv preprint arXiv:2602.02204},
  year={2026}
}

Join the Community

Feel free to ask questions, provide feedbacks and discuss with fellow users of vLLM-Omni in #sig-omni slack channel at slack.vllm.ai or vLLM user forum at discuss.vllm.ai.

Star History

License

Apache License 2.0, as found in the LICENSE file.