VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generation

This repository provides the official implementation of VTBench, a benchmark designed to evaluate the performance of visual tokenizers (VTs) in the context of autoregressive (AR) image generation. VTBench enables fine-grained analysis across three core tasks: image reconstruction, detail preservation, and text preservation, isolating the tokenizer's impact from the downstream generation model.

Our goal is to encourage the development of strong, general-purpose open-source visual tokenizers that can be reliably reused across autoregressive image generation and broader multimodal tasks.

🔥 News

• May 19, 2025: Our paper is now available on arXiv! Read it here
• May 18, 2025: We released the demo on huggingface space! Pick your favorite image and try out over 20+ visual tokenizers. [link]

🔍 Why VTBench?

Recent AR models such as GPT-4o demonstrate impressive image generation quality, which we hypothesize is made possible by a highly capable visual tokenizer. However, most existing VTs significantly lag behind continuous VAEs, leading to:

• Poor reconstruction fidelity
• Loss of structural and semantic detail
• Failure to preserve symbolic information (e.g., text in multilingual images)

VTBench isolates and evaluates VT quality, independent of the downstream model, using standardized tasks and metrics.

✨ Features

• Evaluation on three tasks:
  1. Image Reconstruction (ImageNet, High-Res, Varying-Res)
  2. Detail Preservation (patterns, fine textures)
  3. Text Preservation (posters, academic abstracts, multilingual scripts)
• Supports VTs from models like FlowMo, MaskBiT, OpenMagViT2, VAR, BSQ-ViT, etc.
• Includes baselines from continuous VAEs (e.g., SD3.5L, FLUX.1) and GPT-4o.
• Metrics: PSNR, SSIM, LPIPS, FID, CER, WER
• ✅ Automatic download of all datasets and models -- no manual setup required.

📑 Open-Source Plan

• Huggingface Space Demo
• VTBench arXiv Paper
• Evaluation Code
• Inference Code on Supported VTs
• VTBench Dataset

🚀 Getting Started

1. Clone the repo

git clone https://github.com/huawei-lin/VTBench.git
cd VTBench

2. Install dependencies

conda create -n vtbench python=3.10
conda activate vtbench
pip install -r requirements.txt

3. Select a VT and Run Evaluation

✅ No Manual Downloads Needed
All datasets and models are automatically downloaded during runtime from Hugging Face. You can directly run experiments without manually downloading any files.

📦 Model Zoo

📚 Dataset

VTBench datasets are available on Hugging Face: https://huggingface.co/datasets/huaweilin/VTBench

Run an experiment:

accelerate launch --num_processes=1 main.py \
    --model_name chameleon \
    --dataset_name task3-movie-posters \
    --split_name test \
    --output_dir results \
    --batch_size 4

The script will create the following directory:

results/
├── original_images/
├── reconstructed_images/
└── results/

📊 Evaluate results

python ./evaluations/evaluate_images.py \
    --original_dir results/original_images \
    --reconstructed_dir results/reconstructed_images/ \
    --metrics fid ssim psnr lpips cer wer \
    --batch_size 16 \
    --num_workers 8

ℹ️ Note: cer and wer are only available in text-based reconstruction tasks.

GPT-4o Support

To use GPT-4o for generation:

export OPENAI_API_KEY=${your_openai_key}

🛠️ Automation

We provide automation scripts in examples. Simply run:

bash ./examples/run.sh

For SLURM users, adapt examples/submit.sh accordingly and uncomment the SLURM section in run.sh.

Citation

If you find this project useful, please consider citing:

@article{vtbench,
  author       = {Huawei Lin and
                  Tong Geng and
                  Zhaozhuo Xu and
                  Weijie Zhao},
  title        = {VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generation},
  journal      = {arXiv preprint arXiv:2502.01634},
  year         = {2025}
}