convert_to_quant

Convert safetensors weights to quantized formats (FP8, INT8, NVFP4, MXFP8) with learned rounding optimization for ComfyUI inference.

---

Installation

pip install convert-to-quant

Or install from source:

git clone https://github.com/silveroxides/convert_to_quant.git
cd convert_to_quant
pip install -e .

---

Requirements Summary

[!IMPORTANT] PyTorch must be installed manually with the correct CUDA version for your GPU. This package does not install PyTorch automatically to prevent environment conflicts.

---

Detailed Installation (GPU-Specific)

1. Install PyTorch

Visit pytorch.org to get the correct install command.

Examples:

# CUDA 13.0 (Required for Blackwell NVFP4/MXFP8)
pip install torch --index-url https://download.pytorch.org/whl/cu130

# CUDA 12.8 (Stable)
pip install torch --index-url https://download.pytorch.org/whl/cu128

# CPU only
pip install torch --index-url https://download.pytorch.org/whl/cpu

2. Optional: Triton (needed for INT8)

# Linux
pip install -U triton

# Windows for torch 2.10 and 2.11
pip install -U "triton-windows<3.7"
# Windows for torch 2.12
pip install -U "triton-windows<3.8"

---

Quick Start

Use the command 'ctq -hf' to view arguments for layer exclusion presets for various models

# All examples include metadata and comfy_quant layers for ComfyUI compatible quantization.
# Examples utilize low memory overhead argument to reduce peak RAM/VRAM usage.

# Basic FP8 Tensorcore quantization without learned rounding
ctq -i model.safetensors -o model-fp8mixed.safetensors --comfy_quant --save-quant-metadata --simple --low-memory

# INT8 Row-Wise quantization without learned rounding
ctq -i model.safetensors -o model-int8mixedrow.safetensors --int8 --scaling_mode row --comfy_quant --save-quant-metadata --simple --low-memory

# Blackwell MXFP8 quantization without learned rounding
ctq -i model.safetensors -o model-mxfp8mixed.safetensors --mxfp8 --comfy_quant --save-quant-metadata --simple --low-memory

Use In Code As Module

# Example modular usage of INT8 Row-Wise quantization of Flux2 Klein 9B
from convert_to_quant import quantize

quantize(
    input="./flux-2-klein-9b.safetensors",
    output="./flux-2-klein-9b-int8mixedrow.safetensors",
    comfy_quant=True,
    save_quant_metadata=True,
    verbose="VERBOSE",
    low_memory=True,
    int8=True,
    scaling_mode="row",
    flux2=True,
    simple=True,
    calib_samples=8192
)

Load the output .safetensors file in ComfyUI like any other model.

---

Supported Quantization Formats

INT8 ConvRot

Learned/AdaRound row-wise INT8 ConvRot:

ctq -i model.safetensors -o model-int8-convrot.safetensors --comfy_quant --save-quant-metadata --int8 --scaling-mode row --convrot --convrot-group-size 256

Add --simple to use direct row-wise INT8 quantization without learned/AdaRound optimization:

ctq -i model.safetensors -o model-int8-convrot-simple.safetensors --comfy_quant --save-quant-metadata --int8 --scaling-mode row --convrot --convrot-group-size 256 --simple

Group 256 remains the default when it divides the layer input width. Primary INT8 ConvRot layers otherwise use regular group 64 when compatible, including widths such as 2688.

---

Model-Specific Presets

(See --help-filters for a full list of presets)

---

Key Features

• Learned Rounding: SVD-based optimization minimizes quantization error.
• Bias Correction: Automatic bias adjustment using synthetic calibration data.
• Model-Specific Support: Exclusion lists for sensitive layers (norms, embeddings).
• Three-Tier Quantization: Mix different formats per layer using --custom-layers.

---

Advanced Usage

Exclude Layer Option

Define specific excluded layers with regex patterns for models with no exclusion preset(This is just example):

ctq -i model.safetensors --exclude-layers "(double_blocks.[01]|final_layer|txt_attn.proj)" --comfy_quant

Scaling Modes

# Block-wise scaling for better accuracy
ctq -i model.safetensors --scaling-mode block --block_size 64 --comfy_quant

---

Acknowledgements

Special thanks to:

• Clybius – For Learned-Rounding inspiration.
• lyogavin – For ComfyUI int8_blockwise support.

---

License

MIT License