Qwen3-VL-2B-GRACE-W4G128-AWQ

This repository contains the AWQ-packed INT4 deployment checkpoint of Qwen3-VL-2B-GRACE-W4G128, produced from our GRACE quantization-aware trained Qwen3-VL-2B model.

GRACE (Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs) is an efficient vision-language model training and compression framework. It combines confidence-gated distillation, relational representation alignment, and quantization-aware training to improve the accuracy-efficiency trade-off of compressed vision-language models.

• Paper: GRACE: Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
• Code: ForeverBlue816/GRACE
• Base model: Qwen/Qwen3-VL-2B-Instruct
• Dataset: Lin-Chen/ShareGPT4V
• Collection: GRACE

Model Summary

This checkpoint is intended for deployment and inference. Unlike the QAT training checkpoint, this version stores the model in an AWQ-packed INT4 format, suitable for efficient inference with AWQ-compatible runtimes.

This is the primary GRACE deployment checkpoint. It retains 75.0 average across seven VLM benchmarks, or 98% of the 76.7 GRACE BF16 average. Upload an image in the GRACE-VLM Space, or use the real INT4 command below on an NVIDIA GPU.

[!IMPORTANT] This checkpoint uses the AWQ tensor layout but requires the GRACE loader to rebuild the quantized Qwen3-VL linear layers. A plain Transformers from_pretrained call does not activate this checkpoint's INT4 weights.

About GRACE

GRACE is designed to compress vision-language models while preserving multimodal reasoning ability. The framework introduces three main components:

1. Confidence-Gated Decoupled Knowledge Distillation
   GRACE uses teacher confidence to adaptively control the strength of distillation, reducing the influence of uncertain or noisy teacher predictions.

2. Relational CKA Alignment
   GRACE aligns relational structures between teacher and student visual-token representations using centered kernel alignment.

3. Adaptive Information Bottleneck Controller
   GRACE regulates representation compression through an adaptive information bottleneck objective, improving the balance between compactness and task performance.

For full technical details, please refer to the paper:

Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
Yanlong Chen, Amirhossein Habibian, Luca Benini, Yawei Li
arXiv:2601.22709

Checkpoint Format

This repository contains a packed AWQ deployment checkpoint.

Expected files include:

awq_quantized_modules.json
chat_template.jinja
config.json
generation_config.json
model.safetensors
processor_config.json
tokenizer.json
tokenizer_config.json
README.md

The important files are:

• model.safetensors: AWQ-packed INT4 model weights
• awq_quantized_modules.json: metadata for quantized modules
• config.json: model architecture and quantization configuration
• processor_config.json: multimodal processor configuration
• tokenizer.json and tokenizer_config.json: tokenizer files
• chat_template.jinja: chat template
• generation_config.json: default generation configuration

Difference from the QAT Checkpoint

We release two related W4G128 versions:

The QAT checkpoint stores quantization-aware trained weights and learned quantization information.
This AWQ version is packed into real INT4 deployment tensors for inference.

Quick Start: Real INT4

Docker

docker pull ghcr.io/foreverblue816/grace-vlm:v0.1.0
docker run --platform linux/amd64 --gpus all --rm \
  -v "${HOME}/.cache/huggingface:/root/.cache/huggingface" \
  ghcr.io/foreverblue816/grace-vlm:v0.1.0

The released container targets Linux/amd64 NVIDIA hosts and downloads this checkpoint on first use.

Python environment

git clone https://github.com/ForeverBlue816/GRACE.git
cd GRACE

pip install torch==2.5.1 torchvision==0.20.1 \
  --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements_inference.txt
pip install -e qwen-vl-utils/

python qwen-vl-finetune/scripts/deploy_awq_qwen.py \
  --load-packed ForeverBlue/Qwen3-VL-2B-GRACE-W4G128-AWQ \
  --image deployment/images/chinaairlines.jpg \
  --query "Describe this image in detail."

The loader accepts this Hub repo ID directly and downloads the checkpoint on first use. Install autoawq-kernels to enable the fused CUDA path. Without that optional package, inference remains numerically correct but uses a slower PyTorch dequantization path.

For a standard Transformers from_pretrained experience, use the GRACE BF16 checkpoint.

Recommended Inference Settings

For deterministic evaluation:

generation_kwargs = {
    "max_new_tokens": 256,
    "do_sample": False,
}

For open-ended generation:

generation_kwargs = {
    "max_new_tokens": 512,
    "do_sample": True,
    "temperature": 0.7,
    "top_p": 0.9,
}

Evaluation

This checkpoint is part of the GRACE model family. The GRACE paper evaluates compressed VLMs on a broad set of multimodal benchmarks, including:

• MMBench
• SEED-Bench
• ScienceQA
• HallusionBench
• AI2D
• MMMU
• MMStar

Please refer to the paper and the project repository for the full experimental setup and benchmark results.

Intended Use

This model is intended for research and deployment experiments in:

• efficient vision-language models
• multimodal instruction following
• VLM quantization
• knowledge distillation
• low-bit inference
• edge or resource-constrained multimodal AI systems

Limitations

This checkpoint inherits the limitations of the base Qwen3-VL model and the limitations of low-bit quantized VLMs.

Potential limitations include:

• reduced accuracy compared with full-precision models on difficult reasoning tasks
• sensitivity to prompt formatting and generation settings
• possible hallucinations in visual question answering
• possible OCR or fine-grained perception errors
• possible bias inherited from the base model and training data
• hardware and kernel compatibility issues depending on the AWQ runtime

Users should evaluate the model carefully before using it in high-stakes settings.

Ethical Considerations

This model should not be used for harmful, deceptive, or privacy-invasive applications.
The model may generate inaccurate or biased content, especially when used outside its intended research and deployment context.

For applications involving medical, legal, financial, or safety-critical decisions, human expert review is required.

License

This repository is released under the Apache-2.0 license.

Please also check the licenses and terms of use of the base model and datasets before downstream use.

Citation

If you find this model useful, please cite our paper:

@article{chen2026grace,
  title={Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs},
  author={Chen, Yanlong and Habibian, Amirhossein and Benini, Luca and Li, Yawei},
  journal={arXiv preprint arXiv:2601.22709},
  year={2026}
}

You may also cite the AWQ paper if you use or discuss the AWQ deployment format:

@article{lin2023awq,
  title={AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration},
  author={Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Chen, Wei-Ming and Wang, Wei-Chen and Xiao, Guangxuan and Dang, Xingyu and Gan, Chuang and Han, Song},
  journal={arXiv preprint arXiv:2306.00978},
  year={2023}
}

Acknowledgements

This release builds on the Qwen-VL model family, the Hugging Face Transformers ecosystem, ShareGPT4V, and AWQ-compatible low-bit inference tooling.

We thank the open-source community for making efficient multimodal model research and deployment possible.