vLLM GGUF Quantization Plugin

This plugin provides out-of-tree GGUF quantization support for vLLM after in-tree support deprecation (vllm-project/vllm#39583).

Installation

Prerequisites

• CUDA toolkit or ROCm toolkit

We recommend uv for package management. If you don't have it installed:

curl -LsSf https://astral.sh/uv/install.sh | sh

From Source

1. Clone this repository:

   git clone https://github.com/vllm-project/vllm-gguf-plugin
   cd vllm-gguf-plugin
   

2. If vLLM is not already installed, install it first:

   uv pip install vllm --torch-backend=auto
   

3. Build and install the plugin against the PyTorch installation used by vLLM:

   uv pip install -e . --no-build-isolation
   

   Disabling build isolation ensures that the CUDA extension is compiled against the same PyTorch installation used by vLLM at runtime.

Development

After completing the editable source installation above, install and run the development tooling:

uv pip install -e .[dev] --torch-backend=auto
pre-commit install
pre-commit run --all-files

The same hooks also run in GitHub Actions on every push and pull request.

Usage

vllm serve Qwen/Qwen3-0.6B-GGUF:Q8_0 --tokenizer Qwen/Qwen3-0.6B

Qwen 3.5 MTP speculative decoding loads the nextn block embedded in the same GGUF; it does not download separate Hugging Face MTP weights:

vllm serve unsloth/Qwen3.5-4B-MTP-GGUF:Q4_K_M \
  --tokenizer Qwen/Qwen3.5-4B \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

For a GGUF without a nextn block, omit --speculative-config; the backbone loads normally without MTP.

Tested model coverage

The plugin uses vLLM's model implementations and a generic GGUF weight adapter, so model compatibility is broader than a fixed allowlist. The models below are covered by the repository's generation tests and are the best-known starting points:

Other vLLM-supported architectures may work when their GGUF tensor names map to the corresponding Hugging Face model. A model appearing in vLLM's general supported-model list does not by itself guarantee GGUF compatibility. When reporting an unsupported model, include the model repository, quantization, plugin and vLLM versions, and the complete weight-mapping error.