TPU inference for vLLM, with unified JAX and PyTorch support.
See the code
| Documentation | Blog | User Forum | Developer Slack (#sig-tpu) |
🤝 Contribute to the Project
Looking to help? Click a badge below to find issues that need your attention.
vLLM TPU is now powered by tpu-inference, an expressive and powerful new hardware plugin unifying JAX and PyTorch under a single lowering path within the vLLM project. The new backend now provides a framework for developers to:
Although vLLM TPU’s new unified backend makes out-of-the-box high performance serving possible with any model supported in vLLM, the reality is that we're still in the process of implementing a few core components.
For this reason, we’ve provided Recommended Models and Recommended Features pages detailing the models and features that are validated through unit, integration, and performance testing.
Get started with vLLM on TPUs by following the quickstart guide.
Visit our documentation to learn more.
Compatible TPU Generations
Below is the live status of our supported models, features, and kernels. Click on any category to expand the detailed support table. It is automatically updated from our detailed Support Matrices.
Last Updated: 2026-08-27 07:02 AM UTC
- ✅ Passing: Tested and works as expected. Ready for use.
- ❌ Failing: Known to be broken or not functional. Help is wanted to fix this!
- 🧪 Experimental: Works, but unoptimized or pending community validation.
- 📝 Planned: Not yet implemented, but on the official roadmap.
- ⛔️ Unplanned: There is no benefit to adding this.
- ❓ Untested: The functionality exists but has not been recently or thoroughly verified.
📐 View Matrix Aggregation Rules (v6e/v7x & C+P)
- 🛠️ Correctness + Performance (C + P)
- ❌ Failing: If either check fails.
- ✅ Passing: If BOTH checks pass successfully.
- ❓ Untested: If any check is untested (and neither fails).
- 🌐 Hardware Rollups (v6e + v7x)
- ❌ Failing: If the feature fails on either v6e or v7x.
- ✅ Passing: If the feature passes on BOTH v6e and v7x.
- ❓ Untested: If either generation is untested (and neither fails).
Stable support status for official releases and production deployments.
✅ Tested Models
🚀 Advanced Capabilities
Core Features
Feature Flax Torchax Default async scheduler ✅ ✅ ✅ Chunked Prefill ✅ ✅ ✅ DCN-based P/D disaggregation ✅ ✅ ✅ KV Cache Offload ✅ ✅ ✅ LoRA_Torch ✅ ✅ ✅ Multimodal Inputs ✅ ✅ ✅ Out-of-tree model support ✅ ✅ ✅ Prefix Caching ✅ ✅ ✅ Single Program Multi Data ✅ ✅ ✅ Speculative Decoding: Eagle3 ✅ ✅ ✅ Speculative Decoding: Ngram ✅ ✅ ✅ Speculative Decoding: DFlash ✅ ❓ ✅ hybrid kv cache ❓ ❓ ❓ multi-host ❓ ❓ ❓ runai_model_streamer_loader ❓ ❓ ❓ sampling_params ❓ ❓ ❓ Step Pooling (Embedding) ❓ ❓ ❓ structured_decoding ❓ ❓ ❓ Parallelism Techniques
Feature Flax Torchax Single-host Multi-host Single-host Multi-host PP ✅ ✅ ✅ ✅ DP ✅ ❓ ✅ ❓ EP ✅ ❓ ✅ ❓ TP ✅ ❓ ✅ ❓ CP ❓ ❓ ❓ ❓ SP (vote to prioritize) ❓ ❓ ❓ ❓ Quantization Methods
Checkpoint dtype Method Supported
Hardware AccelerationFlax Torchax FP4 W4A16 mxfp4 v7 ❓ ❓ FP8 W8A16 compressed-tensor v7 ❓ ❓ FP8 W8A8 compressed-tensor v7 ❓ ❓ INT4 W4A16 awq v5, v6 ❓ ❓ INT8 W8A8 compressed-tensor v5, v6 ❓ ❓ NVFP4 W4A16 modelopt_fp4 v7 ❓ ❓ Note:
- This table only tests checkpoint loading compatibility.
🔬 Microbenchmark Kernel Support
Category Test W16A16 W8A8 W8A16 W4A4 W4A8 W4A16 Moe Fused MoE ❓ ❓ ❓ ❓ ❓ ❓ gmm ❓ ❓ ❓ ❓ ❓ ❓ Dense All‑gather matmul ❓ ❓ ❓ ❓ ❓ ❓ Attention Generic Ragged Paged
Attention V3*❓ ❓ ❓ ❓ ❓ ❓ MLA ❓ ❓ ❓ ❓ ❓ ❓ Ragged Paged
Attention V3 Head_Dim
64*❓ ❓ ❓ ❓ ❓ ❓ Note:
- For attention kernels, W[x]A[y] denotes KV cache as W, A as compute, and x, y as bit precision.
Support status for the latest nightly/main branch developments.
✅ Tested Models
Model Type Unit Test Correctness Test Performance Test google/gemma-4-26B-A4B-it Multimodal ✅ ✅ ✅ google/gemma-4-31B-it Multimodal ✅ ✅ ✅ google/gemma-3-27b-it Text ✅ ✅ ✅ meta-llama/Llama-3.1-8B-Instruct Text ✅ ✅ ✅ meta-llama/Llama-3.3-70B-Instruct Text ✅ ✅ ✅ Qwen/Qwen3-30B-A3B Text ✅ ✅ ✅ Qwen/Qwen3-4B Text ✅ ✅ ✅ Qwen/Qwen3-Coder-480B-A35B-Instruct Text ✅ ✅ ✅ Qwen/Qwen3.5-397B-A17B Text ✅ ✅ ✅ Qwen/Qwen2.5-VL-7B-Instruct Multimodal ✅ ✅ ❌ Qwen/Qwen3-Embedding-8B Embedding ✅ ✅ ❓ deepseek-ai/DeepSeek-R1 Text ✅ ✅ ❓ moonshotai/Kimi-K2.6 Text ✅ ✅ ❓ openai/gpt-oss-120b Text ✅ ✅ ❓ google/gemma-4-E2B-it Multimodal ✅ ❌ ❓ google/gemma-4-E4B-it Multimodal ✅ ❌ ❓ Qwen/Qwen3-32B Text ❌ ❓ ❓ deepseek-ai/DeepSeek-OCR Multimodal ❓ ❓ ❓ Qwen/Qwen3-Omni-30B-A3B-Instruct Multimodal ❓ ❓ ❓ Qwen/Qwen3-VL-8B-Instruct Multimodal ❓ ❓ ❓ Qwen/Qwen3.5-9B Multimodal ❓ ❓ ❓ deepseek-ai/DeepSeek-Math-V2 Text ❓ ❓ ❓ deepseek-ai/DeepSeek-V3.1 Text ❓ ❓ ❓ deepseek-ai/DeepSeek-V3.2 Text ❓ ❓ ❓ deepseek-ai/DeepSeek-V3.2-Speciale Text ❓ ❓ ❓ MiniMaxAI/MiniMax-M2.5 Text ❓ ❓ ❓ moonshotai/Kimi-K2-Thinking Text ❓ ❓ ❓ openai/gpt-oss-20b Text ❓ ❓ ❓ zai-org/GLM-5 Text ❓ ❓ ❓ 🚀 Advanced Capabilities
Core Features
Feature Flax Torchax Default async scheduler ✅ ✅ ✅ Chunked Prefill ✅ ✅ ✅ DCN-based P/D disaggregation ✅ ✅ ✅ KV Cache Offload ✅ ✅ ✅ LoRA_Torch ✅ ✅ ✅ Multimodal Inputs ✅ ✅ ✅ Out-of-tree model support ✅ ✅ ✅ Prefix Caching ✅ ✅ ✅ Single Program Multi Data ✅ ✅ ✅ Speculative Decoding: Eagle3 ✅ ✅ ✅ Speculative Decoding: DFlash ✅ ✅ ✅ Speculative Decoding: Ngram ✅ ✅ ✅ hybrid kv cache ❓ ❓ ❓ multi-host ❓ ❓ ❓ runai_model_streamer_loader ❓ ❓ ❓ sampling_params ❓ ❓ ❓ Step Pooling (Embedding) ❓ ❓ ❓ structured_decoding ❓ ❓ ❓ Parallelism Techniques
Feature Flax Torchax Single-host Multi-host Single-host Multi-host PP ✅ ✅ ✅ ✅ DP ✅ ❓ ✅ ❓ EP ✅ ❓ ✅ ❓ TP ✅ ❓ ❌ ❓ CP ❓ ❓ ❓ ❓ SP (vote to prioritize) ❓ ❓ ❓ ❓ Quantization Methods
Checkpoint dtype Method Supported
Hardware AccelerationFlax Torchax FP4 W4A16 mxfp4 v7 ❓ ❓ FP8 W8A16 compressed-tensor v7 ❓ ❓ FP8 W8A8 compressed-tensor v7 ❓ ❓ INT4 W4A16 awq v5, v6 ❓ ❓ INT8 W8A8 compressed-tensor v5, v6 ❓ ❓ NVFP4 W4A16 modelopt_fp4 v7 ❓ ❓ Note:
- This table only tests checkpoint loading compatibility.
🔬 Microbenchmark Kernel Support
Category Test W16A16 W8A8 W8A16 W4A4 W4A8 W4A16 Moe Fused MoE ❓ ❓ ❓ ❓ ❓ ❓ gmm ❓ ❓ ❓ ❓ ❓ ❓ Dense All‑gather matmul ❓ ❓ ❓ ❓ ❓ ❓ Attention Generic Ragged Paged
Attention V3*❓ ❓ ❓ ❓ ❓ ❓ MLA ❓ ❓ ❓ ❓ ❓ ❓ Ragged Paged
Attention V3 Head_Dim
64*❓ ❓ ❓ ❓ ❓ ❓ Note:
- For attention kernels, W[x]A[y] denotes KV cache as W, A as compute, and x, y as bit precision.
We're thrilled you're interested in contributing to the vLLM TPU project! Your help is essential for making our tools better for everyone. There are many ways to get involved, even if you're not ready to write code.
Ways to Contribute:
If you're ready to contribute code, our Contributing Guide is the best place to start. It covers everything you need to know, including:
A huge thank you to everyone who has helped build and improve vllm-project/tpu-inference!
Emoji Contribution Meaning 💻 Code Submitted merged pull requests or code changes. 🐛 Issues Opened valid issues or bug reports. 👀 Reviews Reviewed pull requests and provided feedback.
🏆 Ranking: Contributors are sorted from highest to lowest based on their total effort score (Total Commits + Unique Issues Opened + PRs Reviewed). If there is a tie, contributors are displayed alphabetically.
(top 24 of 28)
889 followers · starred Oct 2025
3,329 followers · starred Apr 2026
400 followers · starred Aug 2026
313 followers · starred Oct 2025
TPU inference for vLLM, with unified JAX and PyTorch support.
See the code
| Documentation | Blog | User Forum | Developer Slack (#sig-tpu) |
🤝 Contribute to the Project
Looking to help? Click a badge below to find issues that need your attention.
vLLM TPU is now powered by tpu-inference, an expressive and powerful new hardware plugin unifying JAX and PyTorch under a single lowering path within the vLLM project. The new backend now provides a framework for developers to:
Although vLLM TPU’s new unified backend makes out-of-the-box high performance serving possible with any model supported in vLLM, the reality is that we're still in the process of implementing a few core components.
For this reason, we’ve provided Recommended Models and Recommended Features pages detailing the models and features that are validated through unit, integration, and performance testing.
Get started with vLLM on TPUs by following the quickstart guide.
Visit our documentation to learn more.
Compatible TPU Generations
Below is the live status of our supported models, features, and kernels. Click on any category to expand the detailed support table. It is automatically updated from our detailed Support Matrices.
Last Updated: 2026-08-27 07:02 AM UTC
- ✅ Passing: Tested and works as expected. Ready for use.
- ❌ Failing: Known to be broken or not functional. Help is wanted to fix this!
- 🧪 Experimental: Works, but unoptimized or pending community validation.
- 📝 Planned: Not yet implemented, but on the official roadmap.
- ⛔️ Unplanned: There is no benefit to adding this.
- ❓ Untested: The functionality exists but has not been recently or thoroughly verified.
📐 View Matrix Aggregation Rules (v6e/v7x & C+P)
- 🛠️ Correctness + Performance (C + P)
- ❌ Failing: If either check fails.
- ✅ Passing: If BOTH checks pass successfully.
- ❓ Untested: If any check is untested (and neither fails).
- 🌐 Hardware Rollups (v6e + v7x)
- ❌ Failing: If the feature fails on either v6e or v7x.
- ✅ Passing: If the feature passes on BOTH v6e and v7x.
- ❓ Untested: If either generation is untested (and neither fails).
Stable support status for official releases and production deployments.
✅ Tested Models
🚀 Advanced Capabilities
Core Features
Feature Flax Torchax Default async scheduler ✅ ✅ ✅ Chunked Prefill ✅ ✅ ✅ DCN-based P/D disaggregation ✅ ✅ ✅ KV Cache Offload ✅ ✅ ✅ LoRA_Torch ✅ ✅ ✅ Multimodal Inputs ✅ ✅ ✅ Out-of-tree model support ✅ ✅ ✅ Prefix Caching ✅ ✅ ✅ Single Program Multi Data ✅ ✅ ✅ Speculative Decoding: Eagle3 ✅ ✅ ✅ Speculative Decoding: Ngram ✅ ✅ ✅ Speculative Decoding: DFlash ✅ ❓ ✅ hybrid kv cache ❓ ❓ ❓ multi-host ❓ ❓ ❓ runai_model_streamer_loader ❓ ❓ ❓ sampling_params ❓ ❓ ❓ Step Pooling (Embedding) ❓ ❓ ❓ structured_decoding ❓ ❓ ❓ Parallelism Techniques
Feature Flax Torchax Single-host Multi-host Single-host Multi-host PP ✅ ✅ ✅ ✅ DP ✅ ❓ ✅ ❓ EP ✅ ❓ ✅ ❓ TP ✅ ❓ ✅ ❓ CP ❓ ❓ ❓ ❓ SP (vote to prioritize) ❓ ❓ ❓ ❓ Quantization Methods
Checkpoint dtype Method Supported
Hardware AccelerationFlax Torchax FP4 W4A16 mxfp4 v7 ❓ ❓ FP8 W8A16 compressed-tensor v7 ❓ ❓ FP8 W8A8 compressed-tensor v7 ❓ ❓ INT4 W4A16 awq v5, v6 ❓ ❓ INT8 W8A8 compressed-tensor v5, v6 ❓ ❓ NVFP4 W4A16 modelopt_fp4 v7 ❓ ❓ Note:
- This table only tests checkpoint loading compatibility.
🔬 Microbenchmark Kernel Support
Category Test W16A16 W8A8 W8A16 W4A4 W4A8 W4A16 Moe Fused MoE ❓ ❓ ❓ ❓ ❓ ❓ gmm ❓ ❓ ❓ ❓ ❓ ❓ Dense All‑gather matmul ❓ ❓ ❓ ❓ ❓ ❓ Attention Generic Ragged Paged
Attention V3*❓ ❓ ❓ ❓ ❓ ❓ MLA ❓ ❓ ❓ ❓ ❓ ❓ Ragged Paged
Attention V3 Head_Dim
64*❓ ❓ ❓ ❓ ❓ ❓ Note:
- For attention kernels, W[x]A[y] denotes KV cache as W, A as compute, and x, y as bit precision.
Support status for the latest nightly/main branch developments.
✅ Tested Models
Model Type Unit Test Correctness Test Performance Test google/gemma-4-26B-A4B-it Multimodal ✅ ✅ ✅ google/gemma-4-31B-it Multimodal ✅ ✅ ✅ google/gemma-3-27b-it Text ✅ ✅ ✅ meta-llama/Llama-3.1-8B-Instruct Text ✅ ✅ ✅ meta-llama/Llama-3.3-70B-Instruct Text ✅ ✅ ✅ Qwen/Qwen3-30B-A3B Text ✅ ✅ ✅ Qwen/Qwen3-4B Text ✅ ✅ ✅ Qwen/Qwen3-Coder-480B-A35B-Instruct Text ✅ ✅ ✅ Qwen/Qwen3.5-397B-A17B Text ✅ ✅ ✅ Qwen/Qwen2.5-VL-7B-Instruct Multimodal ✅ ✅ ❌ Qwen/Qwen3-Embedding-8B Embedding ✅ ✅ ❓ deepseek-ai/DeepSeek-R1 Text ✅ ✅ ❓ moonshotai/Kimi-K2.6 Text ✅ ✅ ❓ openai/gpt-oss-120b Text ✅ ✅ ❓ google/gemma-4-E2B-it Multimodal ✅ ❌ ❓ google/gemma-4-E4B-it Multimodal ✅ ❌ ❓ Qwen/Qwen3-32B Text ❌ ❓ ❓ deepseek-ai/DeepSeek-OCR Multimodal ❓ ❓ ❓ Qwen/Qwen3-Omni-30B-A3B-Instruct Multimodal ❓ ❓ ❓ Qwen/Qwen3-VL-8B-Instruct Multimodal ❓ ❓ ❓ Qwen/Qwen3.5-9B Multimodal ❓ ❓ ❓ deepseek-ai/DeepSeek-Math-V2 Text ❓ ❓ ❓ deepseek-ai/DeepSeek-V3.1 Text ❓ ❓ ❓ deepseek-ai/DeepSeek-V3.2 Text ❓ ❓ ❓ deepseek-ai/DeepSeek-V3.2-Speciale Text ❓ ❓ ❓ MiniMaxAI/MiniMax-M2.5 Text ❓ ❓ ❓ moonshotai/Kimi-K2-Thinking Text ❓ ❓ ❓ openai/gpt-oss-20b Text ❓ ❓ ❓ zai-org/GLM-5 Text ❓ ❓ ❓ 🚀 Advanced Capabilities
Core Features
Feature Flax Torchax Default async scheduler ✅ ✅ ✅ Chunked Prefill ✅ ✅ ✅ DCN-based P/D disaggregation ✅ ✅ ✅ KV Cache Offload ✅ ✅ ✅ LoRA_Torch ✅ ✅ ✅ Multimodal Inputs ✅ ✅ ✅ Out-of-tree model support ✅ ✅ ✅ Prefix Caching ✅ ✅ ✅ Single Program Multi Data ✅ ✅ ✅ Speculative Decoding: Eagle3 ✅ ✅ ✅ Speculative Decoding: DFlash ✅ ✅ ✅ Speculative Decoding: Ngram ✅ ✅ ✅ hybrid kv cache ❓ ❓ ❓ multi-host ❓ ❓ ❓ runai_model_streamer_loader ❓ ❓ ❓ sampling_params ❓ ❓ ❓ Step Pooling (Embedding) ❓ ❓ ❓ structured_decoding ❓ ❓ ❓ Parallelism Techniques
Feature Flax Torchax Single-host Multi-host Single-host Multi-host PP ✅ ✅ ✅ ✅ DP ✅ ❓ ✅ ❓ EP ✅ ❓ ✅ ❓ TP ✅ ❓ ❌ ❓ CP ❓ ❓ ❓ ❓ SP (vote to prioritize) ❓ ❓ ❓ ❓ Quantization Methods
Checkpoint dtype Method Supported
Hardware AccelerationFlax Torchax FP4 W4A16 mxfp4 v7 ❓ ❓ FP8 W8A16 compressed-tensor v7 ❓ ❓ FP8 W8A8 compressed-tensor v7 ❓ ❓ INT4 W4A16 awq v5, v6 ❓ ❓ INT8 W8A8 compressed-tensor v5, v6 ❓ ❓ NVFP4 W4A16 modelopt_fp4 v7 ❓ ❓ Note:
- This table only tests checkpoint loading compatibility.
🔬 Microbenchmark Kernel Support
Category Test W16A16 W8A8 W8A16 W4A4 W4A8 W4A16 Moe Fused MoE ❓ ❓ ❓ ❓ ❓ ❓ gmm ❓ ❓ ❓ ❓ ❓ ❓ Dense All‑gather matmul ❓ ❓ ❓ ❓ ❓ ❓ Attention Generic Ragged Paged
Attention V3*❓ ❓ ❓ ❓ ❓ ❓ MLA ❓ ❓ ❓ ❓ ❓ ❓ Ragged Paged
Attention V3 Head_Dim
64*❓ ❓ ❓ ❓ ❓ ❓ Note:
- For attention kernels, W[x]A[y] denotes KV cache as W, A as compute, and x, y as bit precision.
We're thrilled you're interested in contributing to the vLLM TPU project! Your help is essential for making our tools better for everyone. There are many ways to get involved, even if you're not ready to write code.
Ways to Contribute:
If you're ready to contribute code, our Contributing Guide is the best place to start. It covers everything you need to know, including:
A huge thank you to everyone who has helped build and improve vllm-project/tpu-inference!
Emoji Contribution Meaning 💻 Code Submitted merged pull requests or code changes. 🐛 Issues Opened valid issues or bug reports. 👀 Reviews Reviewed pull requests and provided feedback.
🏆 Ranking: Contributors are sorted from highest to lowest based on their total effort score (Total Commits + Unique Issues Opened + PRs Reviewed). If there is a tie, contributors are displayed alphabetically.
(top 24 of 28)
889 followers · starred Oct 2025
3,329 followers · starred Apr 2026
400 followers · starred Aug 2026
313 followers · starred Oct 2025
