Duo-VLM: Quantized Autoregressive VLM with Semantic Tokenization

Project Overview

This project implements a compact and efficient Vision-Language Model (VLM) for multimodal tasks such as:

• Visually-Grounded Question Answering (VQA)
• Image Captioning
• Image Inpainting (Advanced)

Unlike traditional cross-attention VLMs, this design treats image content as a sequence of discrete semantic visual tokens using a pre-trained tokenizer (e.g., DualViTok). These tokens are processed jointly with natural language by a lightweight Causal Language Model (LLaMA).

---

Key Features & Technology

---

Setup & Installation

✔ Prerequisites

• Python 3.9+
• NVIDIA GPU recommended for training

Clone the Repository

git clone https://github.com/aman050504/Set-Covering-Using-Genetic-Algorithm.git # Replace if different cd Set-Covering-Using-Genetic-Algorithm

Environment Setup

It is recommended to use a virtual environment:

virtual environment setup

pip install virtualenv

virtualenv venv

source venv/bin/activate

Install dependencies:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 pip install transformers peft accelerate bitsandbytes sentencepiece pandas pillow tqdm tensorboard

Dataset

This project supports standard VQA / captioning datasets: Flickr8k MS COCO Captions / VQA

Example Data Structure

/root/nlp/NLP_Project_Data/Flickr8k/ ├── Images/ │ ├── 1000092795.jpg │ └── ... ├── Flickr8k.token.txt └── ...

Training the VLM

Main training script: train_vlm.py Update the following in script before training: LLaMA model path (e.g., meta-llama/LLaMA-3.2-1B) Visual tokenizer checkpoint

Run Training

python train_vlm.py
--data_dir /path/to/Flickr8k
--output_dir /path/to/checkpoints
--epochs 5
--batch_size 4
--learning_rate 2e-4
--qlora_rank 16
--qlora_alpha 32
--grad_clip 1.0 TensorBoard logs are saved in --log_dir.

Inference Inference script: inference_vlm.py

Captioning Example

python inference_vlm.py
--checkpoint_path /path/to/best_model
--image /path/to/image.jpg
--question "Generate a descriptive caption for this image."

VQA Example

python inference_vlm.py
--checkpoint_path /path/to/best_model
--image /path/to/image.jpg
--question "What activity are the children doing?"
--max_tokens 30
--temperature 0.7

Advanced: Image Inpainting

This advanced mode uses the LLM to predict masked visual tokens:

Workflow:

Mask image region → [MASK] token insertion Encode image as visual tokens LLM autoregressively predicts missing tokens Reconstruct final image using decoder

Full implementation is included in: vlm_training_inference.ipynb

References

QLoRA: Low-resource fine-tuning with quantization DualViTok / VQ-GAN style discrete visual encoding