GGUF Quantization support for native ComfyUI models
Python
93
290 commits
updated Oct 16, 2026
GGUF Quantization support for native ComfyUI models including the custom Q8_CR.
[!NOTE]
This is a fork of the original nodes, updated to support loading Ideogram 4 GGUFs and Krea 2 GGUFs. To use this maintained fork, clonehttps://github.com/molbal/ComfyUI-GGUF.
While quantization was previously unfeasible for regular UNET models (conv2d), transformer/DiT models such as flux are less affected by quantization. This allows running them in lower bits per weight variable bitrate quants on GPUs with less VRAM.
More details on how to use it, pre-converted models, and sample workflows are here: Documentation
For technical details on the custom Q8_CR and Q4_CR formats , memory-mapped loading, please see ARCHITECTURE.md.
[!IMPORTANT]
Make sure your ComfyUI is on v0.27.0 or later.
To install the custom node normally, git clone this repository into your custom nodes folder (ComfyUI/custom_nodes) and restart ComfyUI.
git clone https://github.com/molbal/ComfyUI-GGUF
Simply use the GGUF Unet loader found under the bootleg category. Place the .gguf model files in your ComfyUI/models/unet folder.
Pre-quantized models (🍴 icon on ones added by this fork):
Support for quantizing T5 has also been added, these can be used using the various *CLIPLoader (GGUF) nodes which can be used inplace of the regular ones. For the CLIP model, use whatever model you were using before for CLIP. The loader can handle both types of files - gguf and regular safetensors/bin.
mmproj-*.gguf beside the text encoder for image conditioning; text-only workflows do not need it. 🍴This node pack includes a GGUF converter. It has 3 possible interfaces that you can use:
Each option is documented here: Quantizing models
| Format | Storage / execution | Recommended use |
|---|---|---|
| F16 | FP16 GGUF | Maximum compatibility with half-precision storage. |
| BF16 | BF16 GGUF | Preserve BF16 source models where the target supports BF16. |
| Q8_0 | Standard GGML 8-bit | Excellent general-quality 8-bit GGUF. |
| Q5_1 | Standard GGML 5-bit | Lower storage with a quality-oriented 5-bit format. |
| Q5_0 | Standard GGML 5-bit | Lower storage alternative to Q5_1. |
| Q4_1 | Standard GGML 4-bit | Smaller files when VRAM or RAM is constrained. |
| Q4_0 | Standard GGML 4-bit | Smallest supported format for constrained setups. |
| Q6_K | Uniform GGML K 6-bit | High-quality K-quant storage for eligible Linear weights. |
| Q5_K | Uniform GGML K 5-bit | Uniform 5-bit K-quant storage. |
| Q5_K_S / Q5_K_M | Mixed GGML K | Deterministic 5/6-bit distributions; _M protects more sensitive tensors. |
| Q4_K | Uniform GGML K 4-bit | Uniform 4-bit K-quant storage. |
| Q4_K_S / Q4_K_M | Mixed GGML K | Deterministic 4/5/6-bit distributions; _M protects more sensitive tensors. |
| Q8_CR | Per-row INT8 ConvRot | Maintainer recommendation for NVIDIA RTX 30-series systems. |
| Q4_CR | Per-row packed INT4, INT4 activations | Existing W4A4 format; Q4_CR remains its alias. |
| Q4_CR_L | Per-row packed INT4, INT8 activations | Opt-in W4A8 execution using the same packed weight storage. |
K-quant creation uses the GGML-compatible encoder bundled with this converter and requires every selected row to be divisible by 256. Sensitive, convolution, small, and one-dimensional tensors retain higher precision.
For eligible K-quant Linear operations, the experimental bundled Triton/CUDA
backend is enabled by default; unsupported routes continue to use portable
PyTorch dequantization. It accelerates decoding but still materializes a
floating-point weight and uses the same PyTorch Linear as the portable path.
Q3_K is supported on CUDA SM 8.6 with FP16/BF16 activations and up to 8192 rows;
Q5_K/Q6_K remain limited to measured LTX matrix shapes and at most 1024 rows.
Set COMFYUI_GGUF_KQUANT_BACKEND=none before starting ComfyUI to opt out.
See K-quant inference notes for details.
GGUF Quantization support for native ComfyUI models
Python
93
290 commits
updated Oct 16, 2026
GGUF Quantization support for native ComfyUI models including the custom Q8_CR.
[!NOTE]
This is a fork of the original nodes, updated to support loading Ideogram 4 GGUFs and Krea 2 GGUFs. To use this maintained fork, clonehttps://github.com/molbal/ComfyUI-GGUF.
While quantization was previously unfeasible for regular UNET models (conv2d), transformer/DiT models such as flux are less affected by quantization. This allows running them in lower bits per weight variable bitrate quants on GPUs with less VRAM.
More details on how to use it, pre-converted models, and sample workflows are here: Documentation
For technical details on the custom Q8_CR and Q4_CR formats , memory-mapped loading, please see ARCHITECTURE.md.
[!IMPORTANT]
Make sure your ComfyUI is on v0.27.0 or later.
To install the custom node normally, git clone this repository into your custom nodes folder (ComfyUI/custom_nodes) and restart ComfyUI.
git clone https://github.com/molbal/ComfyUI-GGUF
Simply use the GGUF Unet loader found under the bootleg category. Place the .gguf model files in your ComfyUI/models/unet folder.
Pre-quantized models (🍴 icon on ones added by this fork):
Support for quantizing T5 has also been added, these can be used using the various *CLIPLoader (GGUF) nodes which can be used inplace of the regular ones. For the CLIP model, use whatever model you were using before for CLIP. The loader can handle both types of files - gguf and regular safetensors/bin.
mmproj-*.gguf beside the text encoder for image conditioning; text-only workflows do not need it. 🍴This node pack includes a GGUF converter. It has 3 possible interfaces that you can use:
Each option is documented here: Quantizing models
| Format | Storage / execution | Recommended use |
|---|---|---|
| F16 | FP16 GGUF | Maximum compatibility with half-precision storage. |
| BF16 | BF16 GGUF | Preserve BF16 source models where the target supports BF16. |
| Q8_0 | Standard GGML 8-bit | Excellent general-quality 8-bit GGUF. |
| Q5_1 | Standard GGML 5-bit | Lower storage with a quality-oriented 5-bit format. |
| Q5_0 | Standard GGML 5-bit | Lower storage alternative to Q5_1. |
| Q4_1 | Standard GGML 4-bit | Smaller files when VRAM or RAM is constrained. |
| Q4_0 | Standard GGML 4-bit | Smallest supported format for constrained setups. |
| Q6_K | Uniform GGML K 6-bit | High-quality K-quant storage for eligible Linear weights. |
| Q5_K | Uniform GGML K 5-bit | Uniform 5-bit K-quant storage. |
| Q5_K_S / Q5_K_M | Mixed GGML K | Deterministic 5/6-bit distributions; _M protects more sensitive tensors. |
| Q4_K | Uniform GGML K 4-bit | Uniform 4-bit K-quant storage. |
| Q4_K_S / Q4_K_M | Mixed GGML K | Deterministic 4/5/6-bit distributions; _M protects more sensitive tensors. |
| Q8_CR | Per-row INT8 ConvRot | Maintainer recommendation for NVIDIA RTX 30-series systems. |
| Q4_CR | Per-row packed INT4, INT4 activations | Existing W4A4 format; Q4_CR remains its alias. |
| Q4_CR_L | Per-row packed INT4, INT8 activations | Opt-in W4A8 execution using the same packed weight storage. |
K-quant creation uses the GGML-compatible encoder bundled with this converter and requires every selected row to be divisible by 256. Sensitive, convolution, small, and one-dimensional tensors retain higher precision.
For eligible K-quant Linear operations, the experimental bundled Triton/CUDA
backend is enabled by default; unsupported routes continue to use portable
PyTorch dequantization. It accelerates decoding but still materializes a
floating-point weight and uses the same PyTorch Linear as the portable path.
Q3_K is supported on CUDA SM 8.6 with FP16/BF16 activations and up to 8192 rows;
Q5_K/Q6_K remain limited to measured LTX matrix shapes and at most 1024 rows.
Set COMFYUI_GGUF_KQUANT_BACKEND=none before starting ComfyUI to opt out.
See K-quant inference notes for details.